Monthly Archives: July 2005

Quick Oracle: Blink Testing

Posted on by
7

Background:

  1. In testing, an “oracle” is a way to recognize a problem that appears during testing. This contrasts with “coverage”, which has to do with getting a problem to appear. All tests cover a product in some way. All tests must include an oracle of some kind or else you would call it just a tour rather than a test. (You might also call it a test idea, but not a complete test.)
  2. A book called Blink: The Power of Thinking Without Thinking has recently been published on the subject of snap decisions. I took one look at it, flipped quickly through it, and got the point. Since the book is about making decisions based on little information, I can’t believe the author, Malcolm Gladwell, seriously expected me to sit down and read every word.

“Blink testing” represents an oracle heuristic I find quite helpful, quite often. (I used to call it “grokking”, but Michael Bolton convinced me that blink is better. The instant he suggested the name change, I felt he was right.)

What you do in blink testing is plunge yourself into an ocean of data– far too much data to comprehend. And then you comprehend it. Don’t know how to do that? Yes you do. But you may not realize that you know how.

You can do it. I can prove this to you in less than one minute. You will get “blink” in a wink.

Imagine an application that adds two numbers together. Imagine that it has two fields, one for each number, and it has a button that selects random numbers to be added. The numbers chosen are in the range -99 to 99.

Watch this application in action by looking at this movie (which is an interactive EXE packaged in a ZIP file) and ask yourself if you see any bugs. Once you think you have it, click here for my answer.

  • How many test cases do you think that was?
  • Did it seem like a lot of data to process?
  • How did you detect the problem(s)?
  • Isn’t it great to have a brain that notices patterns automatically?

There are many examples of blink testing, including:

  • Page through a long file super rapidly (holding your thumb on the Page Down button, notice the pattern of blurry text on the screen, and look for strange variations in that pattern.
  • Take a 60,000 line log file, paste it into Excel, and set the zoom level to 8%. Scroll down and notice the pattern of line lengths. You can also use conditional formatting in Excel to turn lines red if they meet certain criteria, then notice the pattern of red flecks in the gray lines of text, as you scroll.
  • Flip back and forth rapidly between two similar bitmaps. What catches your eye? Astronomers once did this routinely to detect comets.
  • Take a five hundred page printout (it could be technical documentation, database records, or anything) and flip quickly through it. Ask yourself what draws your attention most about it. Ask yourself to identify three interesting patterns in it.
  • Convert a huge mass of data to sound in some way. Listen for unusual patterns amidst the noise.

All of these involve pattern recognition on a grand scale. Our brains love to do this; our brains are designed to do this. Yes, you will miss some things; no, you shouldn’t care that you are missing some things. This is just one technique, and you use other techniques to find those other problems. We already have test techniques that focus on trees, it also helps to look at the forest.

On Answering Questions About ET

Posted on by
4

People ask me a lot of questions about ET. I want to be helpful in my answers. A problem I struggle with is that questions about ET often come with a lot of assumptions, and the first thing I have to do is to make the assumptions visible and try to clear away the ones that aren’t helpful. Otherwise, my answers will sound crazy.

Questions of any kind rest on premises. That’s cool, and normally it’s not a big problem. It becomes a problem when questions are asked across a paradigmatic chasm. And there’s a big chasm between the premises of “traditional testing” and those of context-driven test methodology, and those of Rapid Software Testing, which is what I call my test methodology.

Starting in 1987, I tried to learn software testing. Starting in 1989, I started reinventing testing for myself, having become disillusioned with the empty calories of folklore that I found in books by folks like William Perry, or the misanthropic techniquism of Boris Beizer (Boris once told me that it didn’t bother him if people find his advice impractical, since he was merely concerned with documenting “best practices”, a phenomenon that he seemed to think has nothing to do with applicability or utility).

I “invented” testing (with the help of many colleagues) mainly by discovering that the problems of testing have already been solved in the fields of cognitive psychology, epistemology, and general systems thinking. The lessons of these much broader and older fields having been studiously ignored by the majority of authors in our field. This puts me in the odd position of having to defend exploratory thinking in technical work as if it’s some kind of new fangled idea, rather than a prime driver of scientific progress since the advent of science itself.

Anyway, now my island of testing metaphysics is mostly complete. I can plan and do and defend my testing without any reference to ideas published in testing “textbooks” or any oral folklore tradition. Instead I reference ideas from logic, the study of cognition, and the philosophy of science. My system works, but it’s a big job to explain it to testing traditionalists, unless they read broadly. For instance, if I were to say that I test the way Richard Feynman used to test, some people get it right away.

Let me illustrate my difficulty: Julian Harty asks “Do you expect an Exploratory Tester to be well versed in [traditional testing] techniques? Do you check that they are competent in them, etc?”

I’ve had some discussions with Julian. He seems like a friendly fellow. My brother Jonathan, who’s had more discussions with him, says “Julian is one of us.” That’s a serious endorsement. So, I don’t want to alienate Julian. I hope I can turn him into an ally.

Still, his question poses a challenge.

Not “exploratory tester”, just “tester.”
First, there is no such thing as an “exploratory tester”, separate from a “traditional tester”, except as a rhetorical device. I sometimes call myself an exploratory tester in debates, by which I mean someone who studies exploratory testing and tries to do it well. But that doesn’t seem to be how Julian is using the term. The truth is all testers are exploratory testers, in that we all test in exploratory ways. Some of us know how to do it well; fewer of us can explain it or teach it.

Testers are testers. Some testers are especially good at simultaneous learning, test design, and test execution, an intellectual blend called exploratory testing.

Exploratory testing is not a technique, it’s an approach.
A technique is a gimmick. A technique is a little thing. There are a buh-zillion techniques. Exploratory thinking is not a technique, but an approach, just as scripted testing is an approach. Approaches modify techniques. Any technique of testing can be approach in an exploratory way or a scripted way, or some combination of the two.

Traditional testing techniques are often not really techniques of testing, they are symbols in a mythology of testing.
Consider the technique “boundary testing.” One would think that this involves analyzing boundaries, somehow, and testing that there are no bugs in software products that are boundary-related. But actually, the way testing is written about and taught, almost no guidance is given to testers about how to analyze anything, including boundaries. Boundary testing isn’t so much a technique as a label, and by repeating the label to each other, we think we are accomplishing something. Now, I do have an exploratory approach to boundary testing. I use various heuristics as part of the boundary testing process, but for the most part, boundary testing is ordinary testing. The technique is a tiny part of it compared to the generic skills of modeling, observing, and evaluating that underlie all skilled testing.

I don’t teach boundary testing in my classes because it’s too trivial to worry about.

So, with that preamble, I can answer the question:

Julian, I assume by “traditional test techniques” you aren’t referring the tradition of using one’s eyes and brain to test something, but rather to certain high sounding labels like equivalence class partitioning (a fancy way of saying “run a different test instead of the same test over and over again”) or black-box testing (a fancy way of saying “test without knowing everything”) or cause-effect graphing (a way of saying “I repeat things I see in books even if the ideas are totally impractical”). I don’t teach those labels to novice testers, Julian, because they don’t help a new tester actually test anything, and I want novices to learn how to test.

But to be an educated tester who is effective at explaining test methodology, I think you need to know the buzzwords; you need to know the folklore. This is true whether you are a tester who embraces exploratory testing, or one who still pretends that you don’t do ET.

A tester– any tester, not just one who follows my rapid testing vision– needs to develop the cognitive skills to effectively question technology. Gain those and you automatically gain everything important about “traditional test techniques”, in my opinion.

To see such a skill in action, ask yourself this question: how many dimensions of a wine glass can you list? Then watch what your mind does next. To answer this question you need a skill that I have come to call “factoring”, which is a component of modeling skill. It is a skill, not a technique, though there may be many techniques we might apply in the course of exhibiting our skill.

Why Talk About Exploratory Testing?

Posted on by
Reply

So, there I was at the Dutch Testing Day, last year. I was a featured speaker, talking about exploratory testing. ET is one of my favorite subjects. It is helpful and powerful, and yet by some strange quirk of history and collective delusion, our industry hasn’t yet embraced it.

They asked me to participate in a panel discussion on ET. But when I took the stage and heard the opening statements by the three other panelists, I realized to my shock that the other guys knew almost nothing about ET. That is to say, each of them seems to have spent ten minutes or so looking at my web site, but indicated no other preparation, background, or study of any kind on this topic.

The panel discussion quickly became a debate. Everybody against me. Part of me loves that. My favorite literary character is Cyrano De Bergerac, and my inner Cyrano delights to be ambushed by brigands on the high road. As a freshman in high school I once was invited to square off alone against the entire senior sociology class, where the issue was whether morality is real or just a convenient human contrivance. Guess which side I was arguing? Hint: Somebody on the other side screamed “Baby killer!” as part of her counterargument.

A lot of things were said. One exchange in particular is intructive:

Panelist: I would never use exploratory testing on a life critical product.

Me: Really? I think it would be irresponsible not to. But let me get this straight. Are you saying that you would NEVER change a test based on something you learned while testing?

Panelist: I change tests often. Everybody does. Is that what you call exploratory testing?

Me: Basically, yes.

Panelist: Well, what does it mean to advocate a practice that everybody already does? That’s like telling us we should breathe.

Me: I’m not advocating that you DO exploratory testing. I’m advocating that you learn to do it WELL. There is a huge difference.

Our poor testing craft is afflicted with diseases. One is testcasetosis, which is the inability to imagine test activities unless packaged in chicklets called test cases. Here I’m concerned with techniquism. That’s the inability to comprehend testing as a skill, but instead only as some set of more or less mechanical behaviors called test techniques.

Exploratory testing is not a technique, it’s an approach. That is to say, any technique can be practiced in an exploratory or non-exploratory way. Exploration itself is not testing, but it modifies testing.

The question “Should I do exploratory testing?” is not helpful. Instead ask “In what way is my testing exploratory and in what way is it scripted? How might my testing benefit from more exploration or more scripting?”

But few people are doing this because exploratory testing is not being discussed. It’s still a closet activity. I go into projects and see lots of ET, but usually no ET is mentioned in any of their officially defined processes.

Come out of the damn closet!

You already do exploratory testing. Learn to see it and talk about it. The constituent skills of exploratory testing are simply the skills of testing, applied in the moment. When you turn the key and your car doesn’t start, the things you do next probably consitute an exploratory testing process.

There’s a lot to this idea of thinking on your feet and changing your approach based on what happens. The games Mastermind, Twenty Questions, Jigsaw puzzles and Sudoku are all exploratory activities.

I will go into more detail and itemize the skills of testing in another post.

— James

Stuart Reid Says “It’s Better Than Nothing”

Posted on by
Reply

I was watching Dr. Stuart Reid talk about model-based testing, some months ago. During the presentation, he complained that so few people used UML for model-making. Why don’t more people use UML, he asked the audience?

I suppose his question was rhetorical, but I couldn’t help myself. I called out “Because it’s clunky!”

“It’s better than nothing,” he replied.

Think about that. Think about that phrase. A comparison is offered between something and nothing. Who could prefer nothing? Nothing is a void. It’s scary and black. It makes us think about death. Ick. But wait a minute, the comparison that matters is not between something and nothing, it’s probably between something and some other thing.

An affliction of people who promote best practices is aversion to alternatives; a passion for monoculture. A desire to have one and only one solution, instead of a spectrum of solutions, to the problems that beset us.

As soon as Stuart said “[constructing state models using the UML formalism] is better than [NOT constructing state models using the UML formalism]” I immediately thought of an alternative which I suppose occupies the “nothing” category in his mind: an alternative to making models in UML is to make models in our heads.

This is not nothing. Our brains are amazing instruments. Don’t pretend that your brain isn’t there or that it does nothing important. Come on, Stuart. I do lots and lots and LOTS of modeling in my head. So do you. A tiny portion of that will we ever commit to any kind record-keeping system.

Another alternative: Pencil and paper. I make little boxes and arrows on paper. It works. Try it.

Yet another alternative: source code. Computer software source code is a modeling language. When I write my software, I am encoding my state models.

What Stuart may have wanted to say is that he believes things would go better in projects if programmers chose to create visual models of software in a specific formal system called UML (to which he may believe there is no viable alternative formal system of visual modeling) prior to writing code, instead of making the leap directly from mental modeling to coding.

It would have been interesting to hear him say that. It’s a debatable sentiment, but at least it’s a way of speaking that doesn’t shut down dialog before it even gets going.

Personally, I find it absurd that a few people would invent an awkward, overly formalized system of making pictures and then hold in contempt the great majority of productive working programmers who choose to ignore it. Remember, just because a consultant wants to take your money doesn’t mean you have to give it him.

— James

No Best Practices

Posted on by
48

Dear Reader,

I would like you to give up, henceforth, the idea of “best practice.” Thank you.

I want to stamp out “best practice” for several reasons:

  1. There are no best practices. By this I mean there is no practice that is better than all other possible practices, regardless of the context. In other words, no matter what the practice and how valuable it may be in one context, I can destroy it by altering things about the situation surrounding the practice.
  2. Although some things that don’t exist can be useful as rhetorical signposts, “best practice” is not one of them. There is nothing honorable you get from pretending that a practice is best that you can’t also get from suggesting that a practice may be good in a specified context, and making a case for that. Sure, there are dishonorable things you can get from “best practice” rhetoric– say, causing a dull-witted client to give you money. If that has tempted you in the past, I urge you to reform.
  3. It has a chilling effect on our progress as an intellectual craft when people pretend that a best practice exists. Best practice blather becomes a substitute for the more difficult, less glamorous, but ultimately more powerful idea of learning how to do your job. By “learning” I mean practicing the skills of identifying and solving patterns of problems we encounter as testers. Of course, that will involve patterns of behavior that we call “practices”. I’m not against the idea of practices, I’m against pretense. Only through pretense can a practice that is interesting in a particular context becomes a “best practice” to which we all must bow down.
  4. When you say that something is a “best practice”, you may impress the uninitiated, or intimidate the inexperienced, but you just look foolish to people who believe in the possibility of excellence. Excellence in an intellectual craft simply cannot be attained by ignorantly copying what other people say that they do. Yet, the notion of a best practice is really just an invitation to be an ignorant pawn in someone else’s game of process manners– or it’s a trick to make people pawns in your own game.
  5. Simple, honest alternatives are available:
    • “Here’s what I would recommend for this situation.”
    • “Here is a practice I find interesting.”
    • “Here is my favorite practice for dealing with {x}.”
    • “{Person X} attributes {practice Y} for his success. Maybe you’d like to learn about it.”

    These alternatives have their own problems, but less so than than does the Oompa Loompa moralizing that lies behind “best practice.” I’m not trying to stamp out all informal communication, I’m just trying to discourage you, dear reader, from making irresponsible recommendations. Saying “best practice” is obviously irresponsible. Recommending a practice more narrowly may also be irresponsible, depending on the situation, but let’s not worry about that right now.

At this point, if you are in the habit of talking about practices that you think everyone should follow, then you must be pretty annoyed with me. You will be annoyed as well if you pin your hope for project success on finding Holy Grail practices that will endow you with grace and prosperity. You may be thinking of certain practices that you think must be best, and getting ready to throw them at me like Pokemon warriors from your collection.

And if you are one of those people who promote a “testing maturity model” then you must be apoplectic. A maturity model is basically a gang of best practices hooked on crystal meth. In my maturity model of the industry, promoting a maturity model is mere level 2 behavior. By level 3, we outgrow it.

So, first let me apologize for raining on your bubble. I have sworn to speak the truth as I see it, but I don’t enjoy this. It doesn’t make me rich, I’ll tell you that. The way to get rich in this world is mainly by making people feel large hope about a small exertion (i.e. “six-second abs”, lottery tickets, voting in an election, maturity models, and stuff like that). If you want to get rich, do not tie yourself to the truth.

Go ahead and follow your favorite practices. Just don’t preach that the rest of us must follow them, too. Keep your process standards to yourself. If you want to make a suggestion, make one that takes context into account.

Here are some common replies to my argument, and my answers:

  • “I don’t mean ‘best practice’ literally. I’m just suggesting that this is a damn good practice.”If you want to say it’s a damn good practice, then it’s just empty hyperbole to call it the BEST practice, right? But the truth is that your practice, whatever it is, is not a damn good practice. The most it can ever be is a damn good practice *within a certain context*. Outside of that context it is something else. So, please define that context before you prescribe it to me.A doctor who prescribes a drug without examining the patient is committing malpractice. Isn’t that obvious? Would you respect such a doctor? Why should we respect people who prescribe practices without regard for the problems faced in a project?The answer comes back, “What if the doctor tells you to eat right and exercise?”And I reply “Even that is irresponsible unless that doctor has seen me. Besides, it’s terribly vague. Ya call that best practice? It’s vacuous blah blah.”
  • “When I said this practice is a ‘best practice’ I meant it was best for a certain context. But pretty much everyone in the industry shares that context, so what’s the point of talking about context?” Often when someone makes this argument to me, I like to learn something about the context they think is so universal, then talk about a particular company or industry segment that doesn’t share that context. This is not difficult.But even if, for the sake of argument, the context were universal, why call it “best”? Maybe this would make sense if there were a world championship competition for practices. If every conceivable practice were brought out and tested against every other one, in some specified context, then maybe there could be some truth to the “best practice” story. Of course, this hasn’t happened, and it can’t happen.
  • “This practice represents a consensus among industry leaders.” No it doesn’t. You’re just making that up. Industry leaders aren’t able to agree on anything much of substance. I know this because I am an industry leader (based on the fact that some people have said so), and other leaders more often disagree with me instead of obeying my commands. I am influential within a certain clique. The software testing industry is made up of many such tiny cliques, each with its own terminology and values.Whenever people tell you that there’s a consensus in the industry, what they probably mean is that there’s an apparent consensus among those people in the industry whom they happen to respect. The best practice for dealing with discordant voices is: just ignore them.(Exercise for the reader: Can you spot the sentence above where I irresponsibly declared a “best practice”? What do you think I really meant by that?)
  • “Lighten up. It’s just an expression.”It’s an expression that cheapens our craft. Please stop using it so that we can build a better craft.

A valued colleague of mine recently sent me a challenge.

“Would you allow that this is a best practice — ‘Seek to understand your problems before selecting solutions for them'”

Here was my reply:

Your advice is generally good, but it is not a universal best practice.

Here’s my analysis.

  1. Do I know what the practice actually is? The practice you specified is pretty vague. What specifically do I do? How do I tell if I’m doing it correctly? It kind of reminds me of this advice: don’t get sick. I would like to follow that advice, but I’m not necessarily able to. I do know to avoid certain behaviors that will probably make me sick; those are the practices. “Remain healthy” or “Understand problems” are goals, not practices.
  2. Does the advice involve a meaningful distinction among possible behaviors? The more you try to specify something that is always best, the more you must avoid making distinctions that might run afoul of context-specific variables. For instance, here’s a potential best practice: Whatever you choose to do that is supposed to have an impact, make sure you do it in this universe, rather than in other universes that might exist in parallel with ours– otherwise what you do won’t have any impact. See? Although I am not able, in the sixty seconds in which I tried, to find an exception to this practice, the fact that there are no exceptions just makes it seem a best practice that isn’t worth discussing. If I have no option but to follow a “practice”, it’s not really a practice, it’s just life. So, if I want to solve a problem, do I have the option of solving it without having understood it? My answer to that, on reflection, is yes. I believe, for some kinds of problems, I can solve them without understanding them. For instance, by staying pretty much in my hotel room when I am in a strange land, I know I must be preempting lots of problems (and therefore in a sense solving them) that might otherwise befall me, even though I can hardly calculate or catalog them, and certainly can’t say that I understand each one.I can think of other examples where there seems to be value in solving a problem that I don’t understand. Here’s one: I saw a news story about something called Dithranol, which has been used to cure psoriasis for a hundred years. Scientists recently announced that yup, it really does cure psoriasis. Up until now they weren’t sure. In other words, somebody apparently solved a problem that they didn’t understand, and that seemed to be okay.Is it possible for me to understand a problem, solve it, and then suffer in some way because I understood the problem? I suspect that some solutions are socially acceptable only because the person doing the solving didn’t know what they were doing at the time. For instance, in California it is illegal for a direct entry midwife to help a woman go through labor, but it is not illegal for a woman to deliver a baby without help, or for her husband or partner to help her deliver. That’s why, officially, I am listed as having delivered my son, even though I paid an illegal crew of midwives to help me. (They were great, BTW.)Now that I’ve identified alternatives, I can ask, is it possible that the alternatives are better than the proposed “best practice”?
  3. Is there a way to measure the quality of the practice? Without a scale and a measuring instrument, the concept of “best” has no meaning. Therefore, I’d like to know how to measure the value of not understanding a problem before I solve it. I don’t have much of a method for doing that. I’m kind of scratching my head about how to assess the value of understanding a problem, versus the value of, say, avoiding understanding. According to dead poet and methodologist Thomas Gray “Where ignorance is bliss, ’tis folly to be wise.” Maybe his is a useful sentiment.
  4. Is there any conceivable situation, then, when I would rationally make the choice to behave in a way opposite to the practice suggested to be “best”? I can think of a few possibilities:
    • If I am trying to score very high on a multiple choice test under time pressure, and I don’t understand one or more of the questions, it may be a better strategy not to interrupt the testing process and seek to understand each question prior to giving the answer. It may be better to give the best answer I can, based on heuristics or cues that come from the structure of the question or the structure of the answers.
    • If I am being tested on my ability to spontaneously draw correct conclusions, under time pressure, the whole point of the test may be for me to select a solution before I consciously understand it.
    • If I am watching a magic show, it would be an insult to the magician if I tried to figure out the trick using all my faculties of understanding the problem. The purpose is to be entertained. Trying to understand too much about the “problem” represented by each trick might spoil the fun.
    • If I am a novice under the supervision of a master, and the master has specified how I am to select solutions in a particular domain. Even though I may not understand the problem, I still may need to follow the instructions as to the selection of solutions. The key thing, here, is that I am not taking responsibility for the quality of the work, only the quality of my following of instructions.

    Do you know what’s behind all this? I think it’s simply that so few of us know how to do our jobs. Like puffer fish, many of us feel that we need to huff ourselves up so that predators won’t devour us. We fluff ourselves full of impressive words. But when I do that I just feel empty.