We. Use. Tools.

Context-Driven testers use tools to help ourselves test better. But, there is no such thing as test automation.

Want details? Here’s the 10,000 word explanation that Michael Bolton and I have been working on for months.

Editor’s Note: I have just posted version 1.03 of this article. This is the third revision we have made due to typos. Isn’t it interesting how hard it is to find typos in your own work before you ship an article? We used automation to help us with spelling, of course, but most of the typos are down to properly spelled words that are in the wrong context. Spelling tools can’t help us with that. Also, Word spell-checker still thinks there are dozens of misspelled words in our article, because of all the proper nouns, terms of art, and neologisms. Of course there are the grammar checking tools, too, right? Yeah… not really. The false positive rate is very high with those tools. I just did a sweep through every grammar problem the tool reported. Out of the five it thinks it found, only one, a missing hyphen, is plausibly a problem. The rest are essentially matters of writing style.

One of the lines it complained about is this: “The more people who use a tool, the more free support will be available…” The grammar checker thinks we should not say “more free” but rather “freer.” This may be correct, in general, but we are using parallelism, a rhetorical style that we feel outweighs the general rule about comparatives. Only humans can make these judgments, because the rules of grammar are sometimes fluid.

“Are you listening? Say something!”

[Note: J. Michael Hammond suggests that I note right at the top of this post that the dictionary definition of listen does not restrict the word to the mere noticing of sounds. In the Oxford English Dictionary, as well as others, an extended and more active definition of listening is also given. It’s that extended definition of listening I am writing about here.]

I’m tired of hearing the simplistic advice about how to listen one must not talk. That’s not what listening means. I listen by reacting. As an extravert, I react partly by talking. Talking is how I chew on what you’ve told me. If I don’t chew on what you say, I will choke or get tummy aches and nightmares. You don’t want me to have nightmares, do you? Until you interrupt me to say otherwise, I charitably assume you don’t.

Below is an alternative theory of listening; one that does not require passivity. I will show how this theory is consistent the “don’t talk” advice if you consider that being quiet while other people speak is one heuristic of good listening, rather than the definition or foundation of it. I am tempted to say that listening requires talking, but that is not quite true. This is my proposal of a universal truth of listening: Listening requires you to change.

To Listen is to Change

  1. I propose that to listen is to react coherently and charitably to incoming information. That is how I would define listening.
  2. To react is to change. The reactions of listening may involve a change of mood, attention, concept, or even a physical action.

Notice that I said “coherently and charitably” and not “constructively” or “agreeably.” I think I can be listening to a criminal who demands ransom even if I am not constructive in my response to him. Reacting coherently is not the same as accepting someone’s view of the world. If I don’t agree with you or do what you want me to, that is not proof of my poor listening. “Coherently” refers to a way of making sense of something by interpreting it such that it does not contradict anything important that you also believe is true and important about the world. “Charitably” refers to making sense of something in a way most likely to fit the intent of the speaker.

Also, notice that coherence does not require understanding. I would not be a bad listener, necessarily, if I didn’t understand the intent or implications of what was told to me. Understanding is too high a burden to require for listening. Coherence and charitability already imply a reasonable attempt to understand, and that is the important part.

Poor listening would be the inability or refusal to do the following:

  • take in data at a reasonable pace. (“reasonable pace” is subject to disagreement)
  • make sense of data that is reasonably sensible in that context, including empathizing with it. (“reasonably sensible” is subject to disagreement)
  • reason appropriately about the data. (“reason appropriately” is subject to disagreement)
  • take appropriate responsibility for one’s feelings about the data (“appropriate responsibility” is subject to disagreement)
  • make a coherent response. (“coherent response” is subject to disagreement)
  • comprehend the reasonable purposes and nature of the interaction (“reasonable purposes and nature” is subject to disagreement)

Although all these elements are subject to disagreement, you might not choose to actively dispute them in a given situation, because maybe you feel that the disagreement is not very important. (As an example, I originally wrote “dispute” in the text above, which I think is fine, but during review, after hearing me read the above, Michael Bolton suggested changing “dispute” to “disagreement” and that seemed okay, too, so I made the change. In making his suggestion, he did not need to explain or defend his preference, because he’s earned a lot of trust with me and I felt listened to.)

I was recently told, in an argument, that I was not listening. I didn’t bother to reply to the man that I also felt he wasn’t listening to me. For the record, I think I was listening well enough, and what the man wanted from me was not listening– he wanted compliance to his world view, which was the very matter of dispute! Clearly he wasn’t getting the reaction he wanted, and the word he used for that was listening. Meanwhile, I had reacted to his statements with arguments against them. To me, this is close to the essence of listening.

If you really believe someone isn’t listening, it’s unlikely that it will help to say that, unless you have a strong personal relationship. When my wife tells me I’m not listening, that’s a very special case. She’s weaker than me and crucial to my health and happiness, therefore I will use every tool at my disposal to make myself easy for her to talk to. I generally do the same for children, dogs, people who seem mentally unstable, fire, and dangerous things, but not for most colleagues. I do get crossed up sometimes. Absolutely. Especially on Twitter. Sometimes I assume a colleague feels powerful, and respond to him that way, only later to discover he was afraid of me.

(This happened again just the other day on Twitter. Which is why it is unlikely you will see me teach in Finland any time soon! I am bitten by such a mistake a few times a year, at least. For me this is not a reason to be softer with my colleagues. Then again, it may be. I struggle with the pros and cons. There is no simple answer. I regularly receive counsel from my most trusted colleagues on this point.)

A Sign of Being Listened to is the Change that Happens

Introspect for a moment. How do you know that your computer is listening to you? At this moment, as I am typing, the letters I want to see are appearing on the screen as I press the keys. WordPress is talking back to me. WordPress is changing, and its changes seem coherent and reasonable to me. My purposes are apparently being served. The computer is listening. Consider what happens when you don’t see a response from your computer. How many times have you clicked “save” or “print” or “calculate” or “paste” and suffered that sinking feeling as the forest noises go completely silent and your screen goes glassy and gets that faraway grayed out look of the damned? You feel out of control. You want to shout at your screen “Come back! I’ve changed my mind! Undo! Cancel!” How do you feel then? You don’t say to yourself “what a good listener my computer is!”

Why is this so? It’s because you are involved in a cybernetic control loop with your computer. Without frequent feedback from your system you lose your control over it. You don’t know what it needs or what to do about it. It may be listening to something, but when nothing changes in a manner that seems to relate to your input, you suspect it is not listening to you.

Based just on this example I conjecture that we feel listened to when a system responds to our utterances and actions in a harmonious manner that honors our purposes. I further conjecture that the advice to maintain attentive silence in order to listen better is a special case of change in such a way as to foster harmony and supportiveness.

Can we think of a situation where listening to someone means shouting loudly over them? I can. I was recently in a situation where a quiet colleague was trying to get students to return to her tutorial after a break. The hallway was too noisy and few people could hear her. I noticed that, so I repeated her words very loudly that her students might hear. I would argue that I listened and responded harmoniously in support of her needs. I didn’t ask her if she felt that I listened to her. She knows I did. I could tell by her smile.

If my wife cries “brake!” when I’m driving, I hit the brake. The physical action of my foot on the brake is her evidence that I listened, not attentive silence or passivity.

It may be a small change or a large change, but for the person communicating with you to feel listened to, they must see good evidence of an appropriate change (or change process) in you.

Let me tell you about being a father of a strong-minded son. I have been in numerous arguments with my boy. I have learned how to get my point across: plant the idea, argue for a while, and then let go of it. I discovered it doesn’t matter if he seems to reject the idea. In fact, I’ve come to believe he cannot reject any idea of mine unless it is genuinely wrong for him. I know he’s listening because he argues with me. And if he gets upset, that means he must be taking it quite seriously. Then I wait. And I invariably see a response in the days that follow (I mean not a single instance of this not happening comes to mind right now).

One of the tragedies of fatherhood is that many fathers can’t tell when their children are listening because they need to see too specific a response too quickly. Some listening is a long process. I know that my son needs to chew on difficult ideas in order to process them. This is how to think about the listening process. True listening implies digestion and incubation. The mental metabolism is subtle, complicated, and absolutely vital.

Let People Chew on Your Ideas

Listening is not primarily about taking information into yourself, any more than eating is about taking food into yourself. With eating the real point is digestion. And for good listening you need to digest, too. Part of digestion is chewing, and for humans part of listening is reacting to the raw data for the purposes of testing understanding and contrasting the incoming data with other data they have. Listening well about any complicated thing requires testing. Does this apply to your spouse and children, too? Yes! But perhaps it applies differently to them than to a colleague at work, and certainly differently than testing-as-listening to politician or a telemarketer.

Why does this matter so much? Because if we uncritically accept ideas we risk falling prey to shallow agreement, which is the appearance of agreement despite an unrecognized deep disagreement. I don’t want to find out in the middle of a critical moment on a project that your definition of testing, or role, or collaboration, or curiosity doesn’t match mine. I want to have conversations about the meanings of words well before that. Therefore I test my understanding. Too many in the Agile culture seem to confuse a vacant smile with philosophical and practical comprehension. I was told recently that for an Agile tester, “collaboration” may be more important than testing skill. That is probably the stupidest thing I have heard all year. By “stupid” I mean willfully refusing to use one’s mind. I was talking to a smart man who would not use his smarts in that moment, because, by his argument, the better tester is the one who agrees to do anything for anyone, not the one who knows how to find important bugs quickly. In other words, any unskilled day laborer off the street, desperate for work, is apparently a better tester than me. Yeah… Right…

In addition to the idea digestion process, listening also has a critical social element. As I said above, whether or not you are listening is, practically speaking, always a matter of potential dispute. That’s the way of it. Listening practices and instances are all tied up in socially constructed rituals and heuristics. And these rituals are all about making ourselves open to reasonable change in response to each other. Listening is about the maintenance of social order as well as maintaining specific social relationships. This is the source of all that advice about listening by keeping attentively quiet while someone else speaks. What that misses is that the speaker also has a duty to perform in the social system. The speaker cannot blather on in ignorance or indifference to the idea processing practices of his audience. When I teach, I ask my students to interrupt me, and I strive to reward them for doing so. When I get up to speak, I know I must skillfully use visual materials, volume control, rhythm, and other rhetorical flourishes in order to package what I’m communicating into a more digestible form.

Unlike many teachers, I don’t interpret silence as listening. Silence is easy. If an activity can be done better and cheaper by a corpse or an inanimate object, I don’t consider it automatically worth doing as a living human.

I strongly disagree with Paul Klipp when he writes: “Then stop thinking about talking and pretend, if it’s not obvious to you yet, that the person who is talking is as good at thinking as you are. You may suddenly have a good idea, or you may have information that the person speaking doesn’t. That’s not a good enough reason to interrupt them when they are thinking.” Paul implies that interrupting a speaker is an expression of dominance or subversion. Yes, it can be, but it is not necessarily so, and I wish someone trained in Anthropology would avoid such an uncharitable oversimplification. Some interruptions are harmful and some are helpful. In fact, I would say that every social act is both harmful and helpful in some way. We must use our judgment to know what to say, how to say it and when. Stating favorite heuristics as if they were “best practices” is patronizing and unnecessary.

One Heuristic of Listening: Stop Talking

Where I agree with Paul and others like him is that one way of improving the harmony of communication and that feeling of being coherently and charitably responded to is to talk less. I’m more likely to use that in a situation where I’m dealing with someone whom I suspect is feeling weak, and whom I want to encourage to speak to me. However, another heuristic I use in that situation is to speak more. I do this when I want to create a rhetorical framework to help the person get his idea across. This has the side effect of taking pressure of someone who may not want to speak at all. I say this based on the vivid personal experience of my first date with the one who would become my wife. I estimate I spoke many thousands of words that evening. She said about a dozen. I found out later that’s just what she was looking for. How do I know? After two dates we got married. We’ve been married 23 years, so far. I also have many vivid experiences of difficult conversations that required me to sit next to her in silence for as long as 10 minutes until she was ready to speak. Both the “talk more” and “talk less” heuristics are useful for having a conversation.

What does this have to do with testing?

My view of listening can be annoying to people for exactly the same reason that testing is annoying to people. A developer may want me to accept his product without “judgment.” Sorry, man. That is not the tester’s way. A tester who doesn’t subject your product to criticism is, in fact, not taking it seriously. You should not feel honored by that, but rather insulted. Testing is how I honor strong, good products. And arguing with you may be how I honor your ideas.

Listening, I claim, is itself a testing process. It must be, because testing is how we come to comprehend anything deeply. Testing is a practice that enables deep learning and deeply trusting what we know.

Are You Listening to Me?

Then feel free to respond. Even if you disagree, you could well have been listening. I might be able to tell from your response, if that matters to you.

If you want to challenge this post, try reading it carefully… I will understand if you skip parts, or see one thing and want to argue with that. Go ahead. That might be okay. If I feel that there is critical information that you are missing, I will suggest that you read the post again. I don’t require that people read or listen to me thoroughly before responding. I ask only that you make a reasonable and charitable effort to make sense of this.


Agile Testing Heuristic: The Power of Looking

Today I broke my fast with a testing exercise from a colleague. (Note: I better not tell you what it is or even who gave it to me, because after you read this it will be spoiled for you, whereas if you read this and at a later time stumble into that challenge, not knowing that’s the one I was talking about, it won’t be spoiled.)

The exercise involved a short spec and an EXE. The challenge was how to test it.

The first thing I checked is if it had a text interface that I could interact with programmatically. It did. So I wrote a program to flood it with “positive” and “negative” input. The results were collected in a log file. I programmatically checked the output and it was correct.

So far this is a perfectly ordinary Agile testing situation. It is consistent with any API testing or systematic domain testing of units you have heard of. The program I wrote performs a check, and the check is produced by my testing thought process and its output analyzed by a similar thought process. That human element qualifies this as testing and not merely naked checking. If I were to hand my automated check to someone else who did not think like a tester, it would not be testing anymore, although the checks would still have some value, probably.

Here’s my public service announcement: Kids! Remember to look at what is happening.

The Power of Looking

One aspect of my strategy I haven’t described yet is that I carefully watched the check as it was running. I do this not as a bored, offhanded, or incidental matter. It’s absolutely vital. I must observe all the output I can observe, rather than just the “pass/fail” status of my checks. I will comb through log files, watch the results in real-time, try things through the GUI, whatever CAN be seen, I want to see it.

As I watched the output flow by in this particular example, I noticed that it was much slower than I expected. Moreover, the speed of the output was variable. It seemed to vary semi-randomly. Since there was nothing in the nature of the program (as I understood it) that would explain slowness or variable timing, this became an instant focus of investigation. Either there’s a bug here or something I need to learn. (Note: that is known as the Explainability Oracle Heuristic.)

It’s possible that I could have anticipated and explicitly checked for performance issues, of course, but my point is that the Power of Looking is a heuristic for discovering lots of things you did NOT anticipate. The models in your mind generate expectations, automatically, that you may not even be aware of until they are violated.

This is important for all testing, but it’s especially important for tool-happy Agile testers, bless their hearts, some of whom consider automation to be next to godliness… Come to think of it, if God has automated his tests for human qualities, that would explain a lot…



Programmer Pairing with a Tester

My sister, Erica, is not a programmer. Normally she’s not a tester, either. But recently she paired with me, playing a tester role, and spotted bugs while I wrote in Perl. In the process, it became clear to me that testers do not need to become programmers in order to help programmers write programs in real-time.

The Context

While working on the report for the Rapid Testing Intensive, recently, I needed a usable archive of the materials. That meant taking all of the pages, comments, and attachments out of my Confluence site and putting them in a form easier to shuffle, subdivide, organize, refer to, and re-distribute. It would be great if that were a feature of Confluence, but the closest I can get to that is either manually downloading each item or downloading an entire archive and dealing with a big abstract blob of XML and cryptically named files with no extensions.

(Note to Atlassian: Please enhance Confluence to include a archivist-friendly (as opposed to system administrator-friendly) archive function that separates pages, attachments, and comments into discrete viewable units with reasonable names.)

The Deflection

While Erica catalogued the names of all the attachments representing student work and the person or persons who created them, I was supposed to write a program to extract the corresponding material from the archive. Instead, I procrastinated. I think I checked email, but I admit it’s possible I was playing Ghost Recon or watching episode 13 of Arang and the Magistrate on Hulu. So, when she was ready with the spreadsheet, I hadn’t even started my program.

To cover my laziness, I thought I’d invite her to sit with me while I wrote it… you know, as if I had been waiting for her on purpose to show her the power of code or whatever. I expected her to decline, since like many computer power users, she has no interest in programming, and no knowledge of it.

The Surprising Outcome

She did not decline. She sat with me and we wrote the program together. She found six or seven important bugs while I typed, and many other little ones. The programming was more fun and social for me. I was more energized and focused. We followed up by writing a second, bigger program together. She told me she wants to do more of this kind of work. We both want to do more.

A Question

How does someone who knows nothing about Perl programming, and isn’t even a tester, productively find bugs almost immediately by looking at Perl code?

That’s kind of a misleading question, because that’s not what really happened. She didn’t just look at my code. She looked at my code in the context of me talking to her about what I was trying to do as I was trying to do it. The process unfolded bit by bit, and she followed the logic as it evolved. It doesn’t take any specific personal quality on the part of the “coding companion,” just general ones like alertness, curiosity, and basic symbolic intelligence. It doesn’t take any particular knowledge, although it can help a lot.

Perhaps this would not work well for all kinds of coding. We weren’t working on something that required heaps of fiddly design, or hours of doodling in a notebook to conceive of some obscure algorithm.

My Claim

A completely non-technical but eager and curious companion can help me write code in real-time by virtue of three things:

  1. The dynamic and interactive legibility of the coding process. I narrate what I’m doing as it comes together step-by-step. The companion doesn’t eat the whole elephant in a bite; and the companion encounters the software mediated by my continuous process of interpretation. I tell him what and why and how. I do this repeatedly, and answer his questions along the way. This makes the process accessible (or in the parlance I like to use “legible” because that word specifically means the accessibility of information). The legibility is not that of a static piece of code, sitting there, but rather a property of something that is changing within a social environment. It’s the same experience as watching a teacher fill a whiteboard with equations. If you came at the end of the class, it would look bewildering, but if you watched it in process, it looks sensible.
  2. The conceptual simplicity of many bugs. Some bugs are truly devious and subtle, but many have a simple essence or an easily recognized form. As I fix my own bugs and narrate that process, my coding companion begins to pick up on regularities and consistency relationships that must be preserved. The companion programs himself to find bugs, as I go.
  3. The sensemaking faculties of a programmer seeking to resolve the confusion of a companion. When my dogs bark, I want to know why they are barking. I don’t know if there’s a good reason or a bad reason, but I want to resolve the mystery. In the course of doing that, I may learn something important (like “the UPS guy is here”). Similarly, when my coding companion says “I don’t understand why you put the dollar sign there and there, but not over there” my mind is directed to that situation and I need to make sense of it. It may be a bug or not a bug, but that process helps me be clear about what I’m doing, no matter what.

And Therefore…

A tester of any kind can contribute early in a development process, and become better able to test, by pairing with a programmer regardless of his own ability to code.

What We Read

I staggered out of the Cambridge Press bookstore a bit dazed, today, having gorged on 21 books. [Addendum: I mean by this that I browsed them, purchased them, and had them shipped home.] If you want to know what a Context-Driven tester reads, here it is:

  • A First Course in Statistical Programming with R
  • Wisdom: Its Nature, Origins, and Development
  • Learning and Expanding with Activity Theory
  • Sequential Analysis and Observational Methods for the Behavioral Sciences
  • Observing Interaction: An Introduction to Sequential Analysis
  • Human Error
  • Combinatorics: A Problem Oriented Approach (Mathematical Association of America Textbooks)
  • A Mathematician Comes of Age (Spectrum)
  • The Golem at Large: What You Should Know about Technology (Canto)
  • The Golem: What You Should Know about Science (Canto)
  • A Practical Introduction to Denotational Semantics (Cambridge Computer Science Texts)
  • Dialogic Inquiry: Towards a Socio-cultural Practice and Theory of Education (Learning in Doing: Social, Cognitive and Computational Perspectives)
  • From Teams to Knots: Activity-Theoretical Studies of Collaboration and Learning at Work (Learning in Doing: Social, Cognitive and Computational Perspectives)
  • Nuts and bolts for the social sciences
  • How to Fold It: The Mathematics of Linkages, Origami and Polyhedra
  • The Cambridge Handbook of Thinking and Reasoning (Cambridge Handbooks in Psychology)
  • The Manuscript of Great Expectations: From the Townshend Collection, Wisbech (Cambridge Library Collection – Literary Studies)
  • The Cognitive Basis of Science
  • Satisficing and Maximizing: Moral Theorists on Practical Reason
  • The Cambridge Handbook of Situated Cognition (Cambridge Handbooks in Psychology)
  • Risk Communication: A Handbook for Communicating Environmental, Safety, and Health Risks

One of the challenges I have for the ISTQB proponents is “What do you read?” You see it’s a trap. If they tell me they read widely, deeply and liberally, I contrast that with the intellectual desert that is the ISTQB Syllabus and ask them why there is such a disconnect between their education and their professional claims. And if they read narrowly, well, there you go.

If you want to be an excellent tester, you need a good education. You didn’t get that in school (or if you’re in school, you’re not getting it), so you need to do something like what I do: scout for fabulous and offbeat books about all the matters of great testing– and testing touches EVERYTHING!

[addendum: If you are not familiar with my distaste for institutional education, before picking a fight with me, go see my book, Secrets of a Buccaneer-Scholar. I spent 26 years doing the research by which I assert that school, although not always destructive and occasionally helpful, is certainly not necessary if you want to live a successful intellectual life. Each day of my life is another data point about how wrong were the teachers who told me I would not be successful without submitting to “the game” of school they desired me to play.]

Most of the books on my list are self-explanatory. One in particular may seem strange: the manuscript of Great Expectations. I picked that one up because the photographic images of Dickens’ original manuscript is a beautiful example of how messy the creative process is. Imagine trying to put metrics on the process of writing that, with all its crossouts and insertions.Writing is exploratory. Just. Like. Testing.

Why Scripted Testing is Not for Novices

…Unless you want bad testing.

Claire Moss writes:

I am surprised that you say that scripted testing is harder for novice
testers. I would have expected that having so much structure around
the tests would make getting into testing easier for someone with less
experience and that the scripted instructions would make up for a lack
of discipline on the part of the tester.

Structure != “being told what to do”
First, you are misusing the word “structure.” All testing is structured. If what you mean by structure is “externally imposed structure” then say that. But even if you are not aware of a structure in your testing, it is there. When I tell a novice tester to test, and don’t tell him how to test, he will be dominated by certain structures he is largely unaware of– or if aware he cannot verbalize or control them much. For instance: the user interface look and feel is a guiding structure for novice testers. They test what they see.

Cognitive science offer plenty of ideas and insights about the structures that guide our thinking and behavior. See the book Predictably Irrational by Dan Ariely for more on this.

Scripted testing always has at least two distinct parts: test design and test execution. They must be considered independently.

Scripted test execution is quite a bit more difficult than exploratory testing, unless you are assuming that the tester following the script has exactly the same knowledge and skill as the test designer (even then it is a qualitatively different sort of cognitive process than designing). An exploratory tester is following (indeed forming as he goes) his own intentions and ideas. But, a scripted tester, to do well, must apprehend the intent of the one who wrote the script. Moreover, the scripted tester must go beyond the stated intent and honor the tacit intent, as well– otherwise it’s just shallow, bad testing.

Try using a script to guide a 10 year-old to drive a car safely on a busy city street. I don’t believe it can be done. You can’t overcome lack of basic skills with written instructions.

And sure, yeah, there is also the discipline issue, but that’s a minor thing, compared to the other things.

As for scripted test design, that also is a special skill. I can ask my son to put together a computer. He knows how to do that. But if I were to ask him for a comprehensive step-by-step set of instructions to allow me to do it, I doubt the result would help me much. Writing a script requires patience, judgment, and lots of empathy for the person who will execute it. He doesn’t yet have those qualities.

Most people don’t like to write. They aren’t motivated. Now give them a task that requires excellent writing. Bad work generally results.

Both on the design side and the execution side, scripted testing done adequately is harder than exploratory testing done adequately. It’s hard to separate an integrated cognitive activity into two pieces and still make it work.

The reason managers assume it’s simpler and easier is that they have low standards for the quality of testing and yet a strong desire for the appearances of order and productivity.

When I am training a new tester, I begin with highly exploratory testing. Eventually, I will introduce elements of scripting. All skilled testers must feel comfortable with scripted testing, for those rare times when it’s quite important.


1. Start browser

2. Go to CNN.com

3. Test CNN.com and report any problems you find.

This looks like a script, and it is sort of a script, but the interesting details of the testing are left unspecified. One of the elements of good test scripting is to match the instructions to the level of the tester as well as to the design goal of the test. In this case, no design goal is apparent.

This script does not necessarily represent bad testing– because it doesn’t represent any testing whatsoever.

1. Open Notepad

2. Type “hello”

3. Verify that “hello” appears on the screen.

This script has the opposite problem. It specifies what is completely unnecessary to specify. If the tester follows this script, he is probably dumbing himself down. There may be some real good reason for these steps, but again, the design goal is not apparent. The tester’s mind is therefore not being effectively engaged. Congratulations, designer, you’ve managed to treat a sophisticated miracle of human procreation, gestation, mothering, socializing, educating, etc. as if he were the equivalent of an animated poking stick. That’s like buying an iPad, then using it as a serving tray for a platter of cheese.

Get Thee to the Konditori

I didn’t see you at the coffee shop, today.

Where were you, tester who told me that you need “certification” because, unlike me, you don’t have a public reputation? Where were you, tester who fears the big machine won’t hire you unless you conform, head bowed, to some lowest common standard? How about you, tester who told me that you dare not stop bullshitting your management about test case counts because “he likes those numbers” and, unlike me, you don’t have the credibility (and therefore the power) to say no without being clapped in chains and thrown into the brig?

I was there as usual, with my hardcover Moleskine and my current favorite pens (the Pigma Micron 01 by Sakura Color Products Corporation and the Mitsubishi Pencil Company Unipin Fineline).

I was working on my intellectual property. What were you doing?

Okay I get it: you were busy with your day job. And when the day is over you are tired and you want to go home. Maybe you have children. Children are important. But don’t tell me you are powerless to change things; that you can’t upgrade yourself. You have the power, but it’s not yet a priority for you. If you have reasonable passion and ambition in this field, you will soar above the problems that so many others feel trapped by. You must take time to develop your craft; educate yourself. Punching the time clock is not enough to obtain a better income and better job security.

One place to start is the Konditori (that’s Swedish for coffee shop… sounds cool and kind of mysterious, don’t you think?).

(This photo, above, is one of two Konditori’s I worked at today, in Stockholm.)

I have to keep developing myself and my intellectual property. It’s the engine by which I feed my family. So, today I worked on Rapid Testing Estimation. I want to roll out my new estimation methodology in time for my next test management class, next week.

(Note: I mostly drink green tea, these days… but it’s Stockholm! You gotta drink coffee in Stockholm!)

But who am I kidding? I work in coffee shops, furiously scribbling in my little notebooks, mainly because I love it. It is practical and effective, sure. Mainly, though, it’s peaceful and… hopeful. I am thrilled to feel that, at any moment, I might experience a breakthrough. And you know those breakthroughs, when they happen, can lead to a new hour or two of cutting edge class material, for which people will pay handsome sums. Last year we had the first conference on Session-based Test Management. My brother and I invented that at a coffee shop (well, it was a Denny’s… that’s pretty close).

We can wipe out bad testing. We can dump commercial certification programs into the dustbin. We can create a powerful craft where testers are well paid and respected. And this is how to do it: Meet me at the Konditori. Bring your Moleskine.

(Only if you love the craft, please. If you don’t, geez man, get a different job!)

Technique: Paired Exploratory Survey

I named a technique the other day. It’s another one of those things I’ve been doing for a while, but only now has come crisply into focus as a distinct heuristic of testing: the Paired Exploratory Survey (PES).

Definition: A paired exploratory survey is a process whereby two testers confront one product at the same time for the purpose of learning a product, preparing for formal testing, and/or characterizing its quality as rapidly as possible, whereby one tester (the “driver”) is responsible for open-ended play and all direct interaction with the product while the other tester (the “navigator” or “leader”) acts as documentarian, mission-minder, and co-test-designer.

Here’s a story about it..

Last week, I was on my way home from the CAST conference with my 17 year-old son Oliver when a client called me with an emergency assignment: “Get down to L.A. and test our product right away!” I didn’t have time to take Oliver home, so we bought some clean clothes, had Oliver’s ID flown in from Orcas Island by bush plane, and headed to SeaTac.

(I love emergencies. They’re exciting. It’s like James Bond, except that my Miss Moneypenny is named Lenore. I got to the airport and two first class tickets were waiting for us. However, a gentle note to potential clients: making me run around like a secret agent can be expensive.)

This is the first time I had Oliver with me while doing professional testing, so I decided to make use of him as an unpaid intern. Basically, this is the situation any tester is in when he employs a non-tester, such as a domain expert, as a partner. In such situations, the professional tester must assure that the non-tester is strongly engaged and having good fun. That’s why I like to make that “honorary tester” drive. I get them twiddling the knobs, punching the buttons, and looking for trouble. Then they’ll say “testing is fun” and help me the next time I ask.

(Oliver is a very experienced video gamer. He has played all the major offline games since he was 3 or 4, and the online ones for the last 5 years. I know from playing with him what this means: he can be relentless once he decides to figure out how a system works. I was hoping his gamer instinct would kick in for this, but I was also prepared for him to get bored and wander off. You shouldn’t set your expectations too high with teenagers.)

The client gave us a briefing about how the device is used. I have already studied up on this, but it’s new for Oliver. The scene reminded me of that part in the movie Inception where Leonardo DiCaprio explains the dynamics of dream invasion.We have a workstation that controls a power unit and connects to a probe which is connected to a pump. It all looks Frankenstein-y.

(I can’t tell you much about the device, in this case. Let’s just say it zaps the patient with “healing energy” and has nothing whatsoever to do with weaponized subconscious projections.)

I set up a camera so that all the testing would be filmed.

(Video is becoming an indispensable tool in my work. My traveling kit consists of a little solid state Sony cam that plugs into the wall so I don’t have to worry about battery life, a micro-tripod so I can pose the camera at any desired angle, and a terabyte hard drive which stores all the work.)

Then, I began the testing just to demonstrate to Oliver the sort of thing I wanted to do. We would begin with a sanity check of the major functions and flows, while letting ourselves deviate as needed to pursue follow-up testing on anything we find that was anomalous. After about 15 minutes, Oliver became the driver, I became the navigator, and that’s how we worked for the next 6 or 7 hours.

Oliver quickly distinguished himself as as a remarkable observer. He noticed flickers on the screen, small changes over time, quirks in the sound the device made. He had a good memory for what he had just been doing, and quickly constructed a mental model of the product.

From the transcript:

“What?!…That could be a problem…check this out…dad…look, right now…settings, unclickable…start…suddenly clickable, during operation…it’s possible to switch its entire mode to something else, when it should be locked!”

and later

“alright… you can’t see the error message every single time because it’s corrupted… but the error message… the error message is exactly what we were seeing before with the sequence bug… the error message comes up for a brief moment and then BOOM, it’s all gone… it’s like… it makes the bug we found with the sequence thing (that just makes it freeze) destructive and takes down the whole system… actually I think that’s really interesting. It’s like this bug is slightly more evolved…”

(You have to read this while imagining the voice of a triumphant teenager who’s just found an easter egg in HALO3. From his point of view, he’s finding ways to “beat the boss of the level.”)

At the start, I frequently took control of the process in order to reproduce the bugs, but as I saw Oliver’s natural enthusiasm and inquisitiveness blossom, I gave him room to run. I explained bug isolation and bug risk and challenged him to find the simplest, yet most compelling form of each problem he uncovered.

Meanwhile, I worked on my notes and noted time stamps of interesting events. As we moved along, I would redirect him occasionally to collect more evidence regarding specific aspects of the evolving testing story.

How is this different from ordinary paired testing?

Paired testing simply means two testers testing one product on the same system at the same time. A PES is a kind of paired testing.

Exploratory testing means an approach to testing whereby learning, test design, and test execution are mutually supportive activities that run in parallel. A PES is exploratory testing, too.

A “survey session,” in the lingo of Session-Based Test Management, is a test session devoted to learning a product and characterizing the general risks and challenges of testing it, while at the same time noticing problems. A survey session contrasts with analysis sessions, deep coverage sessions, and closure sessions, among possible others that aren’t yet identified as a category. A PES is a survey test session.

It’s all of those things, plus one more thing: the senior tester is the one who takes the notes and makes sure that the right areas are touched and right general information comes out. The senior tester is in charge of developing a compelling testing story. The senior tester does that so that his partner can get more engaged in the hunt for vital information. This “hunt” is a kind of play. A delicious dance of curiosity and analysis.

There are lots of ways to do paired testing. A PES is one interesting way.

Hey, I’ve done this before!

While testing with my son, I flashed back to 1997, in one of my first court cases, in which I worked with my brother Jon (who is now a director of testing at eBay, but was then a cub tester). Our job was to apply my Good Enough model of quality analysis to a specific product, and I let Jon drive that time, too. I didn’t think to give a name to that process, at the time, other than ET. The concept of paired testing hadn’t even been named in our community until Cem Kaner suggested that we experiment with it at the first Workshop on Heuristic and Exploratory Techniques in 2001.

I have seen different flavors of a PES, too. I once saw a test lead who stepped to the keyboard specifically because he wanted his intern to design the tests. He felt that that letting the kid lean back in his chair and talk ideas to the ceiling (as he was doing when I walked in) would be the best way to harness certain technical knowledge the intern had which the test lead did not have. In this way, the intern was actually the driver.

I’m feeling good about the name Paired Exploratory Survey. I think it may have legs. Time will tell.

Here’s the report I filed with the client (all specific details changed, but you can see what the report looks like, anyway).

This is What We Do

In the Context-Driven Testing community, the testing craft is a living, growing thing. This dialog, led by my partner in Rapid Testing, Michael Bolton, is a prime example of the life among us. Read the PDF that Michael refers to, and what will you see? You see many ideas proposed and discarded. You see definitions being made, and remade. You see people struggling to make sense of subtle, yet important distinctions.

In my world, the development of testing skill goes hand-in-hand with the development of our rhetoric of describing testing. The development of personal skill is linked to the development of social skill. This is why we smirk and roll our eyes when people come to us looking for templates and other pre-fabricated answers to what they believe are simple questions. My reaction to many who come to me is “You don’t need to learn the definition of term ‘test case’. You don’t need me to tell you ‘how to create a test plan’. What you need is to learn how to test. You need to struggle with imponderables; sit with them; turn them over in your mind. You need practice, and you need to talk through your practice with other testers.”
Michael’s dialog reminds me of the book Proofs and Refutations, by Imre Lakatos, which studies the exploratory and dialectical nature of mathematics by also using dialog.

Shiva is Annoyed with My Questions

A person named Shiva contacted me on Skype back in May. Then he didn’t say anything for several months, until yesterday we had this exchange.


Hi James!




I wanted to set up some time with you to chat about an idea and interest you in it…what would be a good time to chat? Would 1130 AM PST Mon work?


What is it about?


Well, I am on the advisory board of a company in China that does phenomenal work and they are very good in testing and SW dev. wanted to see if you wanted to take advantage of their low rates and high quality and make some money? Especially if you are planning to enter that business?


I run a testing company.


I know.


So… Why would I need a testing company? I already have one!


To improve your margins. Scale.


I don’t see how that would be possible. I do a certain kind of testing that requires a high level of skill. I doubt that any other testing company in the world has that skill. Well, there are a couple, but they are expensive.


That is exactly the point I wanted to walk you through as this company has a phenomenal ability to learn, get resources due to their location and yet do high quality work. It is an idea. I am willing to explore it with you if you are interested. If not, I completely understand.


I would have to see examples of the quality work you say that your company does. Is it posted online? Most companies that say they do quality work don’t do quality work at all. They do terrible work. So, I would need to see what you can do.

Also, I would need to know your training practices. I’d want to see them in writing. If you forward your training materials to me, I could review them.


Sure. Before I go forward, it would be good to have a three-way chat with their president/co-founder, myself and you and I am happy to arrange for samples for you to review – work samples, training practices.


I won’t be interested in talking unless you can show me some basic evidence that we have anything to talk about.

Let me ask you just a few questions:

– Does your test lab document all of its testing in detail? Is every test procedure and action documented?

– Do you have complete expected results documented, too?

– Do you maintain statistics on passed/failed tests? Do you graph them?

– Are your testers ISTQB certified?

Your answers to these questions will allow me to quickly assess your capability. Otherwise, I worry you will waste your time.


James, all these are great questions and I have answers that you will like but I need to think about what you said first. In my opinion, business is not just some Q&A, it is also building cross-company relationships and getting to know each other. I need to think about whether we will be a fit that way at all with each other. Please don’t take this personally but I need to give this some careful thought, or else it may not even work even though we are able to deliver capabilities.


You are right. Business is about relationships. I want a relationship with people who can answer basic questions. These are questions that I routinely answer for my clients.

I’m a testing company, too, right? You know that, right? I know what my clients demand of me, and I’m demanding that of you, too. If what you want is an uncritical client who is easily impressed, you came to the wrong guy.

I’m not taking it personally. I’m taking it as an indication that you are a bit over your head.


There is no need to be rude, James.


I’m not being rude. I’m being honest. It’s not my problem if you can’t handle that.


(Skype indicates Shiva has gone offline.)


It is not rude for a potential customer to challenge your corporate capability. That’s normal due diligence. Your job is to speak honestly and forthrightly about what you can and can’t do. Don’t dodge questions.

The reason I’m skeptical is that almost no test lab actually knows what it is doing. And there’s no excuse. Test labs ought to know how to test, but mostly I see labs much better at faking testing than doing it.

How Challenging Each Other Helps the Craft

Regular readers know that I’m dissatisfied with the state of the testing industry. It’s a shambles, and will continue to be as long as middle managers in big companies continue to be fat juicy targets for scam-artists (large tool vendors, consulting firms, and certain “professional” organizations) and well-meaning cargo cultists (such as those who think learning testing is the same as memorizing definitions of words and filling in templates).

What I can do about it is to develop my personal excellence, and associate myself with others who wish to do likewise. Someday, perhaps we will attain a critical mass. Perhaps the studious will inherit the Earth.

In that spirit, I’m constantly looking for colleagues, and bouncing ideas off of them to make us all better. I challenge people, and to me this is a virtue. It’s how I separate those who will help the craft from those who probably won’t. Sometimes people don’t react well to my challenges. Sometimes that’s because they are bad people (in my estimation); sometimes it’s because they are good people having a bad day; sometimes it’s because I’m having a bad day; or may it’s because I’m a bad person (in their estimation).

Nevertheless, this is a big part of what I do, and I will continue to do it. You have been warned. Also, you have been cheerfully invited to participate.

An Example Challenge and What Came of It:

Lanette Creamer, unlike me, is not a werewolf (though she describes me by the politically correct term “hairy”). If you read her tweets and her blog then you also know she’s, uh, what’s the opposite of brutal? Anyway, I bet she owns at least one calendar featuring pictures of kittens in hilarious costumes.

I met Lanette a few years ago but as I do with most people, I forgot about her (fun fact: I suffer from a mild case of associative prosopagnosia which, for instance, is why I didn’t recognize my own wife consistently until a few months after we were married). Then I met Lanette again at PNSQC, last year, where she made an impression on me as someone easy to talk to. I checked out her blog and liked what I saw.

2009-11-12 11:51:49 jamesmarcusbach: @lanettecream I’ll go look at your blog.

2009-11-12 12:16:23 jamesmarcusbach: Another must read blog for testers. This one by Lanette Creamer (@lanettecream). It’s the attack of the tester ladies. http://bit.ly/3JT65a

2009-11-12 12:19:22 jamesmarcusbach: I think I encouraged @lanettecream to blog a couple of years ago, and then forgot to follow-up. Guess that worked out.

One thing I liked is that she identified herself, on her blog, as being a member of the Context-Driven School of testing. It means that I can reasonably expect such a person to be self-critical and to accept a challenge from me– a leader in that school.

A couple of days later I happened to see a paper Lanette wrote about “reducing test case bloat.” It was sitting on the desk of Matt Osborn while I was visiting him at Microsoft. I flipped through it and found a definition of “test case” that bugged me.

“Clinically defined a test case is an input and an expected result. For my purposes it doesn’t matter if a test case is automated or manual so long as it is a planned test. For the purpose of reducing test case bloat, I’d go further and say that it is a test you plan to execute a minimum of once in the product lifecycle.”

Lanette was referencing the IEEE with her definition. I hate the IEEE definition of test case. If I ever meet the guy who wrote it, I will bite him on the nose. It’s a narrow-minded supercilious idea of test cases, straight from Colonel Blimp. I prefer a broad definition that encompasses the actual field use of the term. For instance: “An instance or variation of a test or test idea.” By my definition, you can point to a list of test ideas, in bullet form, and call them test cases, just like real people at real companies already do, and not be committing a crime against terminology. Also, my definition does not attempt to enforce a specific organization’s notion of what a test must look like. It has to have inputs or it’s not a test? It has to have specific planned expected result? Not when I test, buddy.

I also didn’t like that Lanette presented it as if it were a universally accepted definition. That’s an appeal to authority, which we in the Context-Driven community do our best to avoid.

From Twitter:

2009-11-14 19:40:16 jamesmarcusbach: @lanettecream Yes, listening to the IEEE is fine if you’re not a true student of testing. But people like us ARE the IEEE (or better).

2009-11-14 19:41:10 jamesmarcusbach: @lanettecream I followed the IEEE, too, for a few years, and then realized that whoever came up with those defs wasn’t very thoughtful.

2009-11-14 19:41:44 jamesmarcusbach: @lanettecream Welcome to software testing leadership, where there is no appeal to authority allowed.

2009-11-14 19:43:22 jamesmarcusbach: @lanettecream The reason I bring it up is that I’ve generalized it myself, and I’m curious if your analysis will reveal something new.

2009-11-14 19:45:18 jamesmarcusbach: @lanettecream One way to frame the question: What exactly do you mean by “input?” What exactly do you mean by “expectation?”

2009-11-14 19:46:50 jamesmarcusbach: @lanettecream I think it’s shallow. I think you can do a lot better. Anyway, I’d be interested to see your analysis of that definition.

2009-11-14 19:48:34 jamesmarcusbach: @lanettecream Another way to say it: maybe that definition is okay– but what does it MEAN? Do you know? Have you really thought it through?

2009-11-14 19:50:49 jamesmarcusbach: @lanettecream IEEE is not a person we can cross-examine. It doesn’t think anything. But for the record, it’s totally wrong about planning!

2009-11-14 19:51:22 jamesmarcusbach: @lanettecream That planning stuff is just propaganda. Ask yourself “what does planning MEAN?”

2009-11-14 19:51:57 jamesmarcusbach: @lanettecream They throw around a lot of words without really thinking about them, it seems to me.

2009-11-14 19:52:41 jamesmarcusbach: @lanettecream I can tell you my opinions of all this. But I’d really love to see you blog about it, first. I’m following your blog now.

[sadly, I cannot obtain Lanette’s side of the conversation because Twitter sucks in that particular way…]

I did worry a little bit that Lanette would freak out and think I was attacking her. I’m a little nervous when engaging women this way, especially, since I have more concern about being seen as a big bully. (A man might see me that way, too, but as a fellow man I would have little sympathy. He just has to learn to cope.)

Dialog with Michael Bolton

While waiting to see what Lanette would come up with, I decided to transpect with Michael Bolton on the same topic in the hope that our good natured arguing would help Lanette feel better about the challenge.

James Bach: hey, to help Lanette, could we transpect through IM?
Michael Bolton: Heh.
Michael Bolton: Sure.
James Bach: then I can show her the transcript
Michael Bolton: If you like.  Pray, proceed.

Michael subsequently edited and published the transcript of that conversation.

During that interaction I came up with a thought experiment with which to question the Lanette/IEEE view of test cases. Can you test a clock when you can’t give it input?

The Clock Problem

I have since used this scenario to help explain to my students what I mean by a test.

Lanette’s Response

To my surprise, she wrote two entries. The first one worried me: What Did I Say a Test Case Was?

I went into damage control mode on Twitter…

2009-11-15 03:27:16 jamesmarcusbach: @lanettecream Your post seems a bit defensive. I wasn’t attacking you, I was trying to find out what you meant by what you said.

2009-11-15 03:30:44 jamesmarcusbach: @lanettecream I want to help real testers, too, and when I seek clarity in myself and other testers, it’s because that helps us avoid waste.

2009-11-15 05:09:55 jamesmarcusbach: @lanettecream I feel better hearing that. Questioning you is, from me, a sign of respect. But I don’t mean to push too hard.

2009-11-15 05:22:46 jamesmarcusbach: @lanettecream From your blog, I can tell you are talented. I’m eager to help your talent blossom. One requirement for that is confidence.

2009-11-15 05:25:55 jamesmarcusbach: @lanettecream One source of confidence is to practice working through ideas with your colleagues.

Lanette tried again. Her second post embraced the spirit of my challenge: What is a Test?

Notice how her second post is in the classic form of an exploratory essay. That’s perfect! I wasn’t asking for an ultimate argument and perfect analysis. I was looking for inquiry, insight, and self-examination.

Why should anyone put up with my challenges?

Well, how about career advancement? This can happen in a couple of ways. First, by publicly accepting and responding to my challenge, she improved her reputation for all to see. She shows that she is someone to be taken seriously, because she takes her own learning seriously. Second, she gained the first level of my gratitude and respect, and these things can be redeemed for professional favors of various kinds. When you are part of a community in good standing, you may holler for support and your fellow citizens will turn out in force to help you. When Lanette puts out a question on Twitter, lots of people will try to answer. It’s a great feeling to know you aren’t alone in the industry.

Plus Lanette was later interviewed by uTest. That was partly from how she impressed some of us on Twitter. I also profiled her in my talk on “buccaneer-testers” at Star East.

I hear that Lanette and my brother are collaborating on something together. I’m eager to see what comes out of that.

Another reason people should put up with challenges is that it makes the industry better. We practice our rhetoric and rapid learning. We grow. I’ve said it many times: the major reason all the terrible misconceptions about testing persist after all these years is that there is a worldwide conspiracy among testing writers and consultants not to debate with each other. Live and let live. Don’t rock the boat that feeds you, etc. Yech.

Finally, there’s personal pride. You feel good about yourself when you can take the heat.

When People Run Away

I don’t mind when people say no to a challenge, unless they are claiming to be expert testers. When a consultant or writer in the field won’t engage me, then I have to dismiss him. I can’t take him seriously. Just as I would not expect to be taken seriously if I held myself above the duty of defending my ideas in public. There’s a pretty substantial list of well known people who are professionally non-existent to me, but I don’t know how else to deal with them. We have to have intellectual standards or we can’t get anywhere.

(I know of a couple of exceptions to that rule, both women, whom I won’t name here. They are people who have strong aversions to debate (at least to debating me) and yet have great ideas and have contributed lots of good to the field. I can never be a close colleague of people like that, but I’m glad that they’re out there.)

Remembering Anna Allison

All this reminds me of Anna Allison. She was a rising star in 2001. I had dinner with her after she approached me at a conference and begged for a conversation (anyone can talk to me, at any time, if they give me food). At dinner, she mentioned that she was a bug metrics expert. I rolled my eyes and drew a bug metrics graph, daring her to tell me what it meant. What followed was a tour de force of questioning and analysis. She uncovered every trap that I had put into the graph. I told her she should write an article about our conversation and she did!

Tragically, she was on one of the planes that went into the Twin Towers on 9/11, on her way to a consulting gig in LA. This affected me more than I expected it to, because while I didn’t know her well, personally, professionally she was one of the few people I’ve known for whom debate was great fun. The Context-Driven community lost a happy tigress in her. We need more leaders like that. We really couldn’t spare her when she left us, and no one like her has yet stepped up: a non-threatening personality who is a role model for debate. I think that may be why I have high hopes for Lanette. (Also for Meeta Prakash, BTW.)

What happened yesterday?

Yesterday I issued a challenge to new blogger Michael Alexander. He responded promptly and in admirable fashion.

Lanette subsequently did a video blog about why she reacted to my challenge so constructively.

These events inspired me to explain all this. And so, I call upon all testers to challenge me, challenge yourselves, and challenge each other. Let’s blow out the cobwebs. Let’s be testers, not followers.

Tester Pilot

Richard drove up to the hangar just as I was checking the oil on the Husky, his prized baby float plane. Nuts. He was right on time. I was late. I’m supposed to have the plane ready to go when he arrives.

“Hey Dad, looks like a good day for flying. I’m just in the middle of the pre-flight.”

“Where are we going today?” He asked.

“I haven’t been up for a few months, so I figured just a sightseeing tour around the islands and then some pattern work at Friday Harbor.” I hate pattern work: landing and taking off while talking on the radio to the other pilots. That’s exactly why I need to do it, though. I must get over my nerves; must become a safe pilot. It’s a lesson from testing: focus on the risk.

“How much fuel do we need for that?”

“There’s about 18 or 20 gallons on board. That’s actually enough, but I figure it would be better to bring it to 35, just in case.”

“How much will you put in each tank, then?”

“7 gallons.”

“7 plus 7 plus 18 doesn’t add up to 35. Decide on the fuel you want and get that. If you’re going to fudge, fudge toward having more fuel. What are the four most useless things in the world?”

Oh I know this… “Um, altitude above you… runway behind you… gas in the gas truck, and… Um–”

“–and a tenth of a second ago” he finished. “But you remembered the important one. Gas. We don’t want that terrible feeling when they close the airport for 30 minutes to clean something up on the runway, and we don’t have the fuel to divert.”

I could have quibbled with him. What we would actually do in that situation is land 10 minutes away at Friday Harbor airport, or heck, anywhere, because we’re a float plane in the midst of an archipelago. But that’s not the point. The point was the habit of precision; of being conscious and calculated about the risks I take, wherever reasonable. Again this relates to testing. When I’m testing, the habit of precision comes out in considering and controlling the states of my test platforms and test data, and of knowing what to look for in test results.

Dad called the flight center for a briefing. He already knew what they’d say, since he always checked the weather before he left home, but Richard Bach is an especially fastidious pilot. He’s not exactly “by the book.” It’s more that he prides himself on safety. Nothing surprises him without itself being surprised at how prepared he is for it. Yes he knew you were coming; here’s your cake.

Dad and me

The Tester’s Attitude in the Air

My father’s philosophy dovetails perfectly with the tester’s mindset. We expect things to go wrong. We know we will be surprised at times, but we anticipate the kinds of surprises that might occur. That means we perform checks that might seem unnecessary at times. And we keep our eyes open.

I was almost done with the walkaround when he got off the phone.

“Three knots at zero six zero over at Friday,” he announced.

I paused to visualize, then tried to sound authoritative. “That’s a crosswind favoring runway three four.”

“Yes. We have the same conditions here.”

Cool. I got it right. I’m supposed to pretend to be the pilot. Officially, Dad is the pilot-in-command, but I do everything he would do, while he supervises and is ready to take over in case there’s a problem. While I’m doing the preflight, he’s doing it too, but not telling me anything– unless I miss something important. Each time we fly, I’m determined to find a problem with the aircraft that he hasn’t noticed, first. I haven’t yet succeeded.

“Dad, what’s this rust colored streak coming out of this bolt?” Yay, I found something! “There’s one of each side of the elevator.”

“Just a little bit of rust.” He smiled and materialized a can of WD-40 and blasted the bolts with it. This airplane is pristine, so even a little blemish of rust really stands out.

“Were you flying recently?”

“Yeah, I went out last week and splashed around at Lake Watcom.”

“That explains the streaks. Water spray on the tail. Did you pump out the floats afterward?”

“No, but I doubt there’s more than a pint of water in there.”

“Let’s see about that.” I retrieved the hand pump while he popped out the drain plugs. He was right again, I couldn’t suck out more than a cup of water from the floats, total, from all the compartments.

But there was something odd about the last one.

“This water is PINK, Dad!”

Now he was not smiling.

“Unless you landed at the rainbow lake on Unicorn Planet, there may be a hydraulic leak in there.”

He put his fingers in the residue and sniffed it like a bush guide on the trail of a white tiger. “Yeah, that’s what it is. Let’s pop the hatch and take a look.”

Testing With Open Expectations

This is an example of why good testing is inherently an open investigation. Yes, I had definite ideas of what I was testing for: leaky floats. Yes, I had a specific method in mind for performing that test. Had I not a specific method, I would have developed one on the fly. That’s the testing discipline. My oracles were judging the amount of water I was able to pump out of the floats compared to other occasions, and I also tasted the water a couple of times to detect if it was salty. It shouldn’t be, because it had been several flights since we had landed in salt water, but I check just in case there was a previously undetected leak from before. If salt water gets in there, we could have a serious corrosion problem.

I had no conscious intent to check the color of the water. But in testing we take every anomaly seriously. We take data in and ask ourselves, does this make sense? In that way, we are like secret service agents, trained to deal with known threats and to have a good chance to discern fresh and unusual threats, too.

The question “Can I explain everything that I see?” is a good heuristic to keep in mind while testing.

But if I were to have automated my float pump tests, I would never have found this problem. Because unlike a human, a program can’t look at something and just try to explain it. It must be told exactly what to look for.

I got an email, today…

Geoff confirmed the hydraulic leak at the connection in the left float, and will be sealing it, probably tomorrow.  He’ll move the Husky to the big hangar to do the work. Nice, that you decided to pump the floats!


Free Testing Lessons Over Skype

One of the things I like to do is give testing lessons over Skype. I never charge for them, but there are conditions:

  • I don’t schedule them in advance. I do it as time is available.
  • I coach normally in text form, not by Skype voice, although I sometimes make exceptions.
  • You agree that I may publish all or part of the coaching transcript in an book, article, or blog post, as long as I remove personally identifying details (unless you want to be identified).

Here’s how it works:

1. You contact me over Skype. (ID: SATISFICE)

Contact me and say “I would like to have a coaching session. I have 90 minutes free right now. I have read your coaching policy and it’s okay with me.” Then, if I have time, I do a coaching session with you, right then and there.

2. My Motivation: I am motivated by people who entertain me with their passion for learning. Besides that, I want learn how to coach better. Finally, I’m developing material for another book.

3. The process is Socratic dialog: That means I ask you questions and pose problems, then you develop your answers. I also instruct directly as needed. My goal is to help you develop into a tester worthy of respect according to the standards of the Context-Driven School of Testing.

4. We work on things that matter to you: If you have a goal to learn a specific kind of thing, we work on that. Otherwise, I’m happy to walk you through my own syllabus of testing.

5. I may require you to do homework as a condition of further coaching. I may ask for permission to publish your homework as a training example.

6. I may want to watch you test over Skype screen sharing.

7. You are free to tell others that you are working with me.

8. If you ask me a specific question, I will probably ask you to tell me your answer, first. I will expect you to have googled around and thought about it some before we talk.

You should bear in mind that I’m a demanding tutor. I want to spread clarity and excellence in software testing. Don’t approach me unless you have an ambition to be excellent in your craft.

My offer is open to anyone in the world, as long as I believe I can help you.

(Thank you to Adrian Dinca for informing me how confusing my earlier version of this post was!)

New Version of the ET Dynamics Lists

Michael Bolton, my brother Jon, and I have produced a new version of our Exploratory Testing Dynamics document. We unveiled it last week at STPCON.

This document describes the elements of exploratory testing as we currently understand them. This new version has not yet been reviewed by Cem Kaner or any of our colleagues who normally have opinions on this, but I don’t anticipate that it will be revised much when they do. It should be fairly stable, for now.

ET cannot be summed up fairly as “unstructured playing around” and if you look at this document you will see why. If nothing else, count the elements in it. Crikey.

The document consists of four lists:

1. Evolving Work Products

Exploratory testing is an approach that emphasizes evolution. We start with something and make it better and better. Often testers can be overly focused on finding bugs, but look at all the things that get better. For instance, you can run your automated regression tests a thousand times and the tests don’t learn and neither do the button pushing testers. But you learn a lot by sapiently retesting– you evolve. One of the products of ET is a better tester.

Some evolution also happens in highly scripted testing, but in ET it’s a primary goal. It’s the point.

2. Skills and Tactics

What does an exploratory tester do? Well, feast your eyes on the Skills and Tactics list. Each of those items are things that seem to us to be reasonably independent, teachable, and observable. Some of them are also skills that a highly scripted tester would need, but they work a little differently for explorers. For instance, the skills of reading and analyzing documents means doing that to prepare for or aid exploration of the product, rather than to prepare detailed instructions for scripted test execution.

3. ET Polarities

The Polarities list is unique to ET. An exploratory tester must manage his own mental process, and that process is about developing ideas and using those ideas to search the product for trouble. We’ve found over years of experience doing this that a tester loses steam if he spends too long in one train or type of thought. So, what can he do?

One answer is alternation. We alternate among complementary activities. Two activities are complementary if performing one of them rejuvenates the other. A great example is reading about a product and directly interacting with it.

4. Test Strategy

The Heuristic Test Strategy model is a set of guideword heuristics that we publish as part of the Rapid Software Testing class. In addition to the general ideas of exploration, evolution, and alternation, the guidewords represent the specific subject matter of testing– the conditions we need to succeed, the things we look at, the things we look for, and the specific test techniques we use to produce tests.

With these lists, I can systematically evaluate a tester and identify strengths and weaknesses. We in the Context-Driven School of testing need models like this, because we focus on skills, rather than techniques and tools. This document allows us to focus our efforts and develop specific exercises for tester training.

Sapience and Blowing Peoples’ Minds

I told a rival that I don’t use the term “manual testing” and that I prefer the term “sapient testing” because it’s more to the point. This is evident in the first definition of the word “manual” in the Oxford English Dictionary: 1. a. Of work, an action, a skill, etc.: of or relating to the hand or hands; done or performed with the hands; involving physical rather than mental exertion. Freq. in manual labour. Sapient, on the other hand, means “wise.”

He laughed and said “Bach, you are always making words up.” And then told me that in his opinion manual testing did not evoke the concept of unskilled manual labor. Now, other than establishing that the guy doesn’t have an online account to the O.E.D. (definition of “sweeeeet!” is “an online O.E.D. account.”), or perhaps doesn’t consider dictionaries to be useful sources of information about what words mean, I see something else in his reaction: I blew his mind. What I said doesn’t intersect with anything in his education.

To understand me, the man will have to use his sapience, rather than responding manually (i.e. with his hands).

In other words, I notice that some of my rivals in the testing industry don’t merely disagree with me, they apparently don’t comprehend what I’m saying. Example: after some ten hours of solid debate with me, over several sittings, Stuart Reid (who is working on a software testing standard of all preposterous things), told a colleague of mine that he believed I don’t truly mean the things I said in those debates, but merely said them to “be provocative.” Huh. That’s some serious cognitive dissonance you got going, Stu, when the only way you can process what I’m saying is to declare, essentially, that it was all just a dream.

Of course, I don’t think this is an intelligence problem. I think this is a lack-of-effort-to-use-intelligence problem. It’s not convenient for certain consultants to tear up their out-of-date books and classes and respond to the challenge of, um, the last 30 years of development of the craft. So they continue to teach and preach ideas from the seventies (or create testing standards based on them, because they believe not enough people appreciate testing disco).

Anyway, in the Context-Driven community’s latest attempt to explain the ins and outs and vital subtleties of testing, Michael Bolton has come up with a promising tack. Maybe this will help. He’s drawing a distinction between testing and checking.

Brace yourselves for insight. A lot of what people call testing is actually mere checking. But even checking requires testing intelligence to design and do well. This gives more specifics to my concept of sapient testing. Here are Michael’s seminal posts on the subject:

  1. http://www.developsense.com/2009/08/testing-vs-checking.html
  2. http://www.developsense.com/2009/09/transpection-and-three-elements-of.html
  3. http://www.developsense.com/2009/09/pass-vs-fail-vs-is-there-problem-here.html
  4. http://www.developsense.com/2009/09/elements-of-testing-and-checking.html

When Michael first made the distinction between testing and checking, I was annoyed. Truly. It blew my mind in that bad way. I thought he was manufacturing a distinction that we didn’t need. I decided to ignore it. Then he called me and asked “So what do you think of my checking vs. testing article?” I had to say I didn’t like it at all. We argued…

…and he convinced me that it was a good idea. Thank you dialectical learning! Debate has saved me again!

I now agree that it’s a practical distinction that can be used as a lens to focus on the quality of a test process. I do have to get used to the words, though. I now see a difference between automated testing and automated checking, for instance: automated testing means testing supported by tools; automated checking means specific operations, observations, and verifications that are carried out entirely with tools. Automated testing may INCLUDE automated checking as one component, however automated checking does NOT include testing.

Making this distinction is exactly like distinguishing between a programmer and a compiler. We do not speak of a compiler “writing a program” in assembly language when it compiles C++ code. We do not think that we can fire the programmers because the compiler provides “automated programming.” The same thing goes for testing. Or… does that blow your mind?