- In testing, an “oracle” is a way to recognize a problem that appears during testing. This contrasts with “coverage”, which has to do with getting a problem to appear. All tests cover a product in some way. All tests must include an oracle of some kind or else you would call it just a tour rather than a test. (You might also call it a test idea, but not a complete test.)
- A book called Blink: The Power of Thinking Without Thinking has recently been published on the subject of snap decisions. I took one look at it, flipped quickly through it, and got the point. Since the book is about making decisions based on little information, I can’t believe the author, Malcolm Gladwell, seriously expected me to sit down and read every word.
“Blink testing” represents an oracle heuristic I find quite helpful, quite often. (I used to call it “grokking”, but Michael Bolton convinced me that blink is better. The instant he suggested the name change, I felt he was right.)
What you do in blink testing is plunge yourself into an ocean of data– far too much data to comprehend. And then you comprehend it. Don’t know how to do that? Yes you do. But you may not realize that you know how.
You can do it. I can prove this to you in less than one minute. You will get “blink” in a wink.
Imagine an application that adds two numbers together. Imagine that it has two fields, one for each number, and it has a button that selects random numbers to be added. The numbers chosen are in the range -99 to 99.
Watch this application in action by looking at this movie (which is an interactive EXE packaged in a ZIP file) and ask yourself if you see any bugs. Once you think you have it, click here for my answer.
- How many test cases do you think that was?
- Did it seem like a lot of data to process?
- How did you detect the problem(s)?
- Isn’t it great to have a brain that notices patterns automatically?
There are many examples of blink testing, including:
- Page through a long file super rapidly (holding your thumb on the Page Down button, notice the pattern of blurry text on the screen, and look for strange variations in that pattern.
- Take a 60,000 line log file, paste it into Excel, and set the zoom level to 8%. Scroll down and notice the pattern of line lengths. You can also use conditional formatting in Excel to turn lines red if they meet certain criteria, then notice the pattern of red flecks in the gray lines of text, as you scroll.
- Flip back and forth rapidly between two similar bitmaps. What catches your eye? Astronomers once did this routinely to detect comets.
- Take a five hundred page printout (it could be technical documentation, database records, or anything) and flip quickly through it. Ask yourself what draws your attention most about it. Ask yourself to identify three interesting patterns in it.
- Convert a huge mass of data to sound in some way. Listen for unusual patterns amidst the noise.
All of these involve pattern recognition on a grand scale. Our brains love to do this; our brains are designed to do this. Yes, you will miss some things; no, you shouldn’t care that you are missing some things. This is just one technique, and you use other techniques to find those other problems. We already have test techniques that focus on trees, it also helps to look at the forest.
Meetu Makhija says
This is explained very nicely. A comprehensive article to go through. Without the exe it may have been difficult to understand what blink test is !!
I came across this post while reading Matthew Heusser’s blog! This is a very powerful demonstration on how to use our brain to quickly find bugs in a program. The movie is very interactive. Honestly speaking, I could not have caught the 2 bugs in the software so quickly without using blink testing. Thanks James for showing me yet another way of testing software.
Jesse Dyer says
So what were the bugs, folks? I caught two – the program was selecting values over 99, and the top field never selected negatives. Anything else?
[James’ Reply: Those are the two I know about. James Lyndsay also reported that the field was blank, once, but that may have been an artifact of the recording system I used.]
Rajesh Maadireddy says
An another way of doing testing and thinking testing and teaching testing.
Without knowing what the product is for . We can’t play or have fun with the application.
Gabinet kosmetyczny Szczecin says
We have to always use brains, machines cant do everything for us 🙂
Uridah Sami Ahmed says
Wow, This was interesting! Never considered the possibility of explaining something using an exe 😀
Brilliant article and thank you for the demonstration in your .exe file.
I also noticed that the random number generator never produced a 0 from what I could see.