I’ll do almost anything for free food.

September

• Reston, Virginia
• San Diego, California

October

• Stockholm, Sweden
• Tartu, Estonia
• Cluj-Napoca, Romania
• Bucharest, Romania

(The event in Cluj-Napoca is a one-day public seminar that introduces my Rapid Testing methodology through a series of puzzle challenges and lecture. I will also take any and all questions about testing. Click here to sign up for that.)

November

• Tallinn, Estonia

December

• England
• (tentatively) Singapore

(I have one class to teach in England. I’m interested in doing something more, if there is any interest. Email me.)

My full schedule is published here.

For me test is a verb. Testing is something that I do, not so much something that I create. Testing is the act of exploration of an unknown territory. It is casting questions, like Molotov cocktails, into the darkness, where they splatter and burst into bright revealing fire.

How to Manage Such a Process?

My brother Jon and I created a way to control highly exploratory testing 10 years ago, called session-based test management (SBTM). I recently returned from an intense testing project in Israel, where I used SBTM. But I also experimented with a new idea: thread-based test management (TTM).

Like many of my new ideas, it’s not really new. It’s the christening (with words) and sharpening (with analysis) of something many of us already do. The idea is this: organize management around threads of activity rather than test sessions or artifacts.

Thread-based testing is a generalized form of session-based testing, in that sessions are a form of thread, but a thread is not necessarily a session. In SBTM, you test in uninterrupted blocks of time that each have a charter. A charter is a mission for that session; a light sort of commitment, or contract. A thread, on the other hand, may be interrupted, it may go on and on indefinitely, and does not imply a commitment. Session-based testing can be seen as an extension of thread-based testing for especially high accountability and more orderly situations.

I define a thread as a set of one or more activities intended to solve a problem or achieve an objective. You could think of a thread as a very small project within a project.

Why Thread-Based Test Management?

Because it can work under even the most chaotic and difficult conditions. The only formalism required for TBTM is a list of threads. I use this form of test management when I am dropped into a project with as little a day or two to get it done.

What Does Thread-Based Test Management Looks Like?

It’s simple. Thread-based test management looks like a todo list, except that we organize the todo items into an outline that matches the structure of the testing process. Here’s a mocked-up example:

Test Facilities

• Power meter calibration method
• Backup test jig validation
• Create standard test images

Test Strategy

• Accuracy Testing
  • Sampling strategy
  • Preliminary-testing
  • Log file analysis program
• Transaction Flow Testing
• Essential Performance Testing
• Safety Testing
  • warnings and errors FRS review
  • tool for forcing errors
• Compliance Testing
• Test Protocol V1.0 doc.

Test Management

• Change protocol definition
• Build protocol definition
• Test cycle protocol definition
• Bug reporting protocol definition
• Bug triage
• Fix verifications

This outline describes the high level threads that comprise the test project. I typically use a mind-mapping program like MindManager to organize and present them.

So, you should be thinking, “Is that it? Todo lists?” right about now. Well, no. That’s not it. But that’s one face of it.

What Else Does Thread-Based Test Management Look Like?

It looks like testers gathered around a todo list, talking about what they are going to work on that afternoon. Then they split up and go to work. Several times day they might come together like that. If the team is not co-located, then this meeting is done over instant messaging, email, or perhaps through a wiki.

Is That All it Looks Like?

Well, there is also the status report. Whether written or spoken, the thread-based test management version of a status report lists the threads, who is working on the threads, and the outlook for each thread. It typically also includes an issues list.

Other documentation may be produced, of course. TBTM doesn’t tell you what documents to create. It simply tells you that threads are the organizing principle we use for managing the project.

Where Do Threads Come From?

Threads are first spawned from our model of the testing problem. The Satisfice Heuristic Test Strategy Model is an example of such a model. By working through those lists, we get an idea of the kinds of testing we might want to do: those are the first of the threads. After that, threads might be created in many ways, including splitting off of existing threads as we gain a deeper understanding of what testing needs to be done. Of course, in an Agile environment, each user story kicks off a new testing thread.

Which Threads Do We Work On?

Think priority and progress. We might frequently drop threads, switch threads, and pick them up again. In general, we work on the highest priority threads, but we also work on lower priority threads many times, when we see the possibility for quick and inexpensive progress. If I’m trying to finish a sanity check on the new build, I might interrupt that to discuss the status of a particular known bug if the developer happens to wander by.

Major ongoing threads often become attached to specific people. For instance “client testing” or “performance testing” often become full-time jobs. Testing itself, after all, can be thought of as a thread so challenging to do well, and so different from programming, that most companies have seen fit to hire dedicated testers.

How Do Threads End?

A thread ends either in a cut or knot. Cutting a thread means to cancel that task. A knot, however, is a milestone; an achievement of some kind. This is exactly the meaning of the phrase “tying up the loose ends” and marks either the end of the thread (or group of threads) or a good place to drop it for a while.

How Do We Estimate Work?

In thread-based test management, there is no special provision or method for estimating work, except that this is done on a thread-by-thread basis. Session-based test management may be overlaid onto TBTM in order to estimate work in terms of sessions.

How Do We Evaluate Progress?

In thread-based test management, there is no special provision or method for evaluating progress, either, except that this is done on a thread-by-thread basis, and status reports may be provided frequently, perhaps at the end of each day. Session-based test management is also helpful for that.

So What?

This form of management is actually quite common. But, to my knowledge, no one has yet named and codified it. Without a convenient way to talk about it, we have a hard time explaining and justifying it. Then when the “process improvement” freaks come along, they act like there’s no management happening at all. This form of management has been “illegible” up to now (meaning that it’s there but no one notices it) and my brother and I are going to push to make it fully legible and respectable in the testing arena.

From now on, when asked about my approach to test management, I can say “I practice Rapid Testing methodology, which I track in either a thread-based or session-based manner, depending on the stage of the project, and in a risk-based manner at all times.”

How is TBTM Any Different From Using a TO-DO List?

Michel Kraaij questions the substance of TBTM by wondering how it’s different from the age-old idea of a “to-do” list? See his post here.

This is a good question. Yes, TBTM is different than just using a to-do list, but even so, I don’t think I’ve ever read an article about to-do list based test management (TDBTM?). Most textbooks focus on artifacts, not the activity of testing. Thread-based test management is trying to capture the essence of managing with to-do lists, plus some other things in addition to that.

The main additional element, beyond just making a to-do list, is that a traditional to-do list contains items that can be “done”, whereas many threads might not ever be “done.” They might be cut (abandoned) or knotted (temporarily parked at some level of completion). Some threads maybe tied up with a bow and “done” like a normal task, but not the main ones that I’m thinking of. As I practice testing, for instance, I’m rarely “done” with test strategy. I tinker with the test strategy all the way through the project. That’s why it makes sense to call it a thread.

Another thing to recognize is that the main concern of TBTM is how to know what to put on your thread list. The answer to that invokes the entire framework of Rapid Software testing. So, yeah, it’s more than having an outline of threads, which does look very much like a to-do list– it’s the activity (and skills) of making the list and managing it. If you want to talk about to-do list based test management, then you would have to invent that lore as well. You couldn’t just say “make a to-do list” and claim to have communicated the methodology.

[You can find Jonathan's take on TBTM here.]

[I credit Sagi Krupetski, the test lead on my recent project, for helping me get this idea. His clockwork status reporting and regular morning question "Where are we on the project and what do you think you need to work on today?" caused me to see the thread structure of our project clearly for the first time. He's back on the market now (Chicago area), in case you need a great tester or test manager.]

A non-exploratory doctor is known as… “irresponsible.”
A non-exploratory welder is known as… “irresponsible.”
A non-exploratory pilot is known as… “killed in a plane crash.”
A non-exploratory writer known as… “a plagiarist.”
A non-exploratory scientist is known as… “a tobacco company scientist.” (also “Creationist”)
A non-exploratory truck driver is known as… “lost.”

Can you spot the pattern here?

There is no such thing as an “exploratory tester” except inasmuch as a good tester obviously can and will do exploration as a basic part of his work.

Consider Raisin Bran cereal. If you lived in America and weren’t in solitary confinement during the 80’s an 90’s you would have seen this commercial for Raisin Bran at some point (or one like it):

Two scoops of raisins!

Huh? Two scoops of raisins? What does that mean?

Perhaps the conversation went like this:

“I want my Bran to have MANY raisins!” barked Boss Kellogg.

“But, Mr. Kellogg, we already include nearly one full standard scoop.” replied the Chief Cereal Mixer. “No one has more raisins than we do.”

“Increase to maximum scoop!”

“But sir! that would violate every–”

“TWO SCOOPS! And damn the consequences.”

“The skies will be black with raisins!”

“Then we shall eat in the shade.”

I doubt anything like that happened, though. I suspect what happened is that somebody mixed some raisins with some bran flakes until it tasted pretty good. Maybe he adjusted it a little to optimize cost of goods (and perhaps they adjust the bran/raisin ratio as cost of goods change). Later, I bet, and completely unrelated to the engineering and manufacturing process, Kellogg’s advertising agency decided to create the impression that customers are getting a lot of value for the money, so they invented a distinguishing characteristic that actually makes no sense at all: an absolute measurement called a “scoop”. And began to speak of it AS IF it were meaningful.

The reason the measurement makes no sense at all is that the “Two Scoops” slogan was pasted onto boxes of substantially different sizes. But even if the measurement makes no sense, the pretentious claim makes a lot of sense, because we humans don’t think through the rational basis of measurements like this unless we are A) rather well trained, and more importantly B) highly motivated. So our unconscious lizard brain says to itself “two means yummy. two means yummy. means two yummy. yummy two…”

At some point, someone (an intern, perhaps) may have asked “But are there actually two scoops of raisins in those boxes?” and the answer was much laughing. Because it could be argued that if there are at least two raisins in the box, then there are two scoops of raisins in the box. It could be argued that if there is one raisin in the box and you used two scoops to measure it (”measure twice and cut once”) then there are two scoops of raisins in the box. If you make up your own measuring unit, such as, say, “scoop”, you can go on to make any other claim you want. This is exactly the point of Jerry Weinberg’s famous dictum “If quality doesn’t matter, you can achieve any other goal you want.”

I was thinking about doing a scientific analysis of this, but someone beat me to it.

Oh What Silliness… OR IS IT?

We have a real problem in testing, and no good solution for it. We are supposed to report the ground truth. Concrete reality. But this turns out to be a very difficult matter. Apart from all the problems of observation and interpretation, we have to summarize our findings. When we do that we are tempted to use scientific tropes (such as nonsensical measurements)  to bolster our reports, even when they are poorly founded. We are often encouraged to do this by managers who were raised on Kelloggs commercials and therefore confuse numbers with food.

Let’s look once again at the Raisin Bran situation and consider what might be the reasonable communication hidden there:

Maybe “two scoops” is intended to mean “ample” or “amply supplied with raisins.” In other words they are saying “You won’t regret buying our Raisin Bran, which always has enough raisins for you. While you’re eating it, we predict you will hum the ‘two scoops of raisins!’ song instead of calling a lawyer or becoming a cereal killer.”

I think there’s a scale built into all of us. It’s a comparative scale. It goes like this:

• Minimum Possible
• Nothing
• Hardly any
• Some
• Enough
• Plenty
• Remarkable
• “OMG! That must be a record!”
• Maximum Possible

This scale is a bit of a mess. The italicized values move around (e.g. maximum possible may be not enough in some situation). The others although fixed relative to each other, aren’t fixed in any way more definite than their ordering. The scale is highly situational. It’s relative to our understanding of the situation. For instance you might be impressed to learn that the Colonia cable ship, which was the largest cable ship in the world in 1925, could carry 300 miles of cable in her hold. If so you would be very easily impressed, because I just lied to you… According to that article it actually could hold 3,000 miles of cable. (However, bonus points if you were thinking “what KIND of cable?”)

What I do with bug numbers, etc.

I want you to notice my first paragraph in this post. Notice that every sentence in that paragraph invokes an unspecified quantity.

• “I have often…” Often compared to what?
• “Lots of…” Lots compared to what?
• “Very…” Very compared to what?
• “Vital…” Vital compared to what?

You could say “He’s not saying anything definite in those sentences.” I agree, I’m not. I’m just giving an impression. My point is this: an impression is a start. An impression might be reasonable. An impression may make conversation possible. An impression may make conversation successful.

Most engineering statements like this don’t stand alone. Like flower buds, they blossom under the sunlight of questioning. And that’s why I can’t take any engineer seriously who gets offended when his facts are questioned. They cry: “Don’t you believe me?” I answer: “I don’t know what you mean, so belief has no meaning, yet.”

So, as a professional tester who prides himself on self-examination, I am ready for the probing perspective question that might follow my attempt to send an impression: “compared to what?” I am ready for the data question, too: “what did you see or hear that leads you to say this?”

I strive (meaning I consciously and consistently work on this) to be reasonable and careful in my use of qualifiers, quantifiers, quantities, and intensifiers. For instance, you will notice that I just used the word “reasonable”, by which I intend to invoke images of normal professional practice in your mind (A LOT like invoking the image of two healthy reasonable scoops of delicious raisins).

One important and definite thing that is accomplished by this relatively loose use of language is that it allows us to talk to each other without bogging down the conversation with ALL the specifics RIGHT NOW.

Kelloggs used the method mostly to trick you into buying their bran smothered raisin products. They didn’t have any reasoning behind “two scoops.” But we can use the same technique wisely and ethically, if we choose. We can be ready to back up our claims.

For Bugs: If I tell you I “found X bugs!!” in your product, the number of exclamation points indicates the true message. An exclamation point means “remarkable” or “lots.” If I tell you I found a lot of bugs in your product, I mean I found substantially more than I expected to find in the product, and more than a reasonable and knowledgeable person in this situation would consider acceptable. And by “more” I don’t mean quantity of bug reports, I mean the totality of diversity of problems, impact of problems, and frequency of occurrence of problems. The headline for that is “lots of bugs” or maybe I should say “two scoops of bugs!”

What I can do about it is to develop my personal excellence, and associate myself with others who wish to do likewise. Someday, perhaps we will attain a critical mass. Perhaps the studious will inherit the Earth.

In that spirit, I’m constantly looking for colleagues, and bouncing ideas off of them to make us all better. I challenge people, and to me this is a virtue. It’s how I separate those who will help the craft from those who probably won’t. Sometimes people don’t react well to my challenges. Sometimes that’s because they are bad people (in my estimation); sometimes it’s because they are good people having a bad day; sometimes it’s because I’m having a bad day; or may it’s because I’m a bad person (in their estimation).

Nevertheless, this is a big part of what I do, and I will continue to do it. You have been warned. Also, you have been cheerfully invited to participate.

An Example Challenge and What Came of It:

Lanette Creamer, unlike me, is not a werewolf (though she describes me by the politically correct term “hairy”). If you read her tweets and her blog then you also know she’s, uh, what’s the opposite of brutal? Anyway, I bet she owns at least one calendar featuring pictures of kittens in hilarious costumes.

I met Lanette a few years ago but as I do with most people, I forgot about her (fun fact: I suffer from a mild case of associative prosopagnosia which, for instance, is why I didn’t recognize my own wife consistently until a few months after we were married). Then I met Lanette again at PNSQC, last year, where she made an impression on me as someone easy to talk to. I checked out her blog and liked what I saw.

2009-11-12 11:51:49 jamesmarcusbach: @lanettecream I’ll go look at your blog.

2009-11-12 12:16:23 jamesmarcusbach: Another must read blog for testers. This one by Lanette Creamer (@lanettecream). It’s the attack of the tester ladies. http://bit.ly/3JT65a

2009-11-12 12:19:22 jamesmarcusbach: I think I encouraged @lanettecream to blog a couple of years ago, and then forgot to follow-up. Guess that worked out.

One thing I liked is that she identified herself, on her blog, as being a member of the Context-Driven School of testing. It means that I can reasonably expect such a person to be self-critical and to accept a challenge from me– a leader in that school.

A couple of days later I happened to see a paper Lanette wrote about “reducing test case bloat.” It was sitting on the desk of Matt Osborn while I was visiting him at Microsoft. I flipped through it and found a definition of “test case” that bugged me.

“Clinically defined a test case is an input and an expected result. For my purposes it doesn’t matter if a test case is automated or manual so long as it is a planned test. For the purpose of reducing test case bloat, I’d go further and say that it is a test you plan to execute a minimum of once in the product lifecycle.”

Lanette was referencing the IEEE with her definition. I hate the IEEE definition of test case. If I ever meet the guy who wrote it, I will bite him on the nose. It’s a narrow-minded supercilious idea of test cases, straight from Colonel Blimp. I prefer a broad definition that encompasses the actual field use of the term. For instance: “An instance or variation of a test or test idea.” By my definition, you can point to a list of test ideas, in bullet form, and call them test cases, just like real people at real companies already do, and not be committing a crime against terminology. Also, my definition does not attempt to enforce a specific organization’s notion of what a test must look like. It has to have inputs or it’s not a test? It has to have specific planned expected result? Not when I test, buddy.

I also didn’t like that Lanette presented it as if it were a universally accepted definition. That’s an appeal to authority, which we in the Context-Driven community do our best to avoid.

From Twitter:

2009-11-14 19:40:16 jamesmarcusbach: @lanettecream Yes, listening to the IEEE is fine if you’re not a true student of testing. But people like us ARE the IEEE (or better).

2009-11-14 19:41:10 jamesmarcusbach: @lanettecream I followed the IEEE, too, for a few years, and then realized that whoever came up with those defs wasn’t very thoughtful.

2009-11-14 19:41:44 jamesmarcusbach: @lanettecream Welcome to software testing leadership, where there is no appeal to authority allowed.

2009-11-14 19:43:22 jamesmarcusbach: @lanettecream The reason I bring it up is that I’ve generalized it myself, and I’m curious if your analysis will reveal something new.

2009-11-14 19:45:18 jamesmarcusbach: @lanettecream One way to frame the question: What exactly do you mean by “input?” What exactly do you mean by “expectation?”

2009-11-14 19:46:50 jamesmarcusbach: @lanettecream I think it’s shallow. I think you can do a lot better. Anyway, I’d be interested to see your analysis of that definition.

2009-11-14 19:48:34 jamesmarcusbach: @lanettecream Another way to say it: maybe that definition is okay– but what does it MEAN? Do you know? Have you really thought it through?

2009-11-14 19:50:49 jamesmarcusbach: @lanettecream IEEE is not a person we can cross-examine. It doesn’t think anything. But for the record, it’s totally wrong about planning!

2009-11-14 19:51:22 jamesmarcusbach: @lanettecream That planning stuff is just propaganda. Ask yourself “what does planning MEAN?”

2009-11-14 19:51:57 jamesmarcusbach: @lanettecream They throw around a lot of words without really thinking about them, it seems to me.

2009-11-14 19:52:41 jamesmarcusbach: @lanettecream I can tell you my opinions of all this. But I’d really love to see you blog about it, first. I’m following your blog now.

[sadly, I cannot obtain Lanette's side of the conversation because Twitter sucks in that particular way...]

I did worry a little bit that Lanette would freak out and think I was attacking her. I’m a little nervous when engaging women this way, especially, since I have more concern about being seen as a big bully. (A man might see me that way, too, but as a fellow man I would have little sympathy. He just has to learn to cope.)

Dialog with Michael Bolton

While waiting to see what Lanette would come up with, I decided to transpect with Michael Bolton on the same topic in the hope that our good natured arguing would help Lanette feel better about the challenge.

James Bach: hey, to help Lanette, could we transpect through IM?
Michael Bolton: Heh.
Michael Bolton: Sure.
James Bach: then I can show her the transcript
Michael Bolton: If you like.  Pray, proceed.

Michael subsequently edited and published the transcript of that conversation.

During that interaction I came up with a thought experiment with which to question the Lanette/IEEE view of test cases. Can you test a clock when you can’t give it input?

I have since used this scenario to help explain to my students what I mean by a test.

Lanette’s Response

To my surprise, she wrote two entries. The first one worried me: What Did I Say a Test Case Was?

I went into damage control mode on Twitter…

2009-11-15 03:27:16 jamesmarcusbach: @lanettecream Your post seems a bit defensive. I wasn’t attacking you, I was trying to find out what you meant by what you said.

2009-11-15 03:30:44 jamesmarcusbach: @lanettecream I want to help real testers, too, and when I seek clarity in myself and other testers, it’s because that helps us avoid waste.

2009-11-15 05:09:55 jamesmarcusbach: @lanettecream I feel better hearing that. Questioning you is, from me, a sign of respect. But I don’t mean to push too hard.

2009-11-15 05:22:46 jamesmarcusbach: @lanettecream From your blog, I can tell you are talented. I’m eager to help your talent blossom. One requirement for that is confidence.

2009-11-15 05:25:55 jamesmarcusbach: @lanettecream One source of confidence is to practice working through ideas with your colleagues.

Lanette tried again. Her second post embraced the spirit of my challenge: What is a Test?

Notice how her second post is in the classic form of an exploratory essay. That’s perfect! I wasn’t asking for an ultimate argument and perfect analysis. I was looking for inquiry, insight, and self-examination.

Why should anyone put up with my challenges?

Well, how about career advancement? This can happen in a couple of ways. First, by publicly accepting and responding to my challenge, she improved her reputation for all to see. She shows that she is someone to be taken seriously, because she takes her own learning seriously. Second, she gained the first level of my gratitude and respect, and these things can be redeemed for professional favors of various kinds. When you are part of a community in good standing, you may holler for support and your fellow citizens will turn out in force to help you. When Lanette puts out a question on Twitter, lots of people will try to answer. It’s a great feeling to know you aren’t alone in the industry.

Plus Lanette was later interviewed by uTest. That was partly from how she impressed some of us on Twitter. I also profiled her in my talk on “buccaneer-testers” at Star East.

I hear that Lanette and my brother are collaborating on something together. I’m eager to see what comes out of that.

Another reason people should put up with challenges is that it makes the industry better. We practice our rhetoric and rapid learning. We grow. I’ve said it many times: the major reason all the terrible misconceptions about testing persist after all these years is that there is a worldwide conspiracy among testing writers and consultants not to debate with each other. Live and let live. Don’t rock the boat that feeds you, etc. Yech.

Finally, there’s personal pride. You feel good about yourself when you can take the heat.

When People Run Away

I don’t mind when people say no to a challenge, unless they are claiming to be expert testers. When a consultant or writer in the field won’t engage me, then I have to dismiss him. I can’t take him seriously. Just as I would not expect to be taken seriously if I held myself above the duty of defending my ideas in public. There’s a pretty substantial list of well known people who are professionally non-existent to me, but I don’t know how else to deal with them. We have to have intellectual standards or we can’t get anywhere.

(I know of a couple of exceptions to that rule, both women, whom I won’t name here. They are people who have strong aversions to debate (at least to debating me) and yet have great ideas and have contributed lots of good to the field. I can never be a close colleague of people like that, but I’m glad that they’re out there.)

Remembering Anna Allison

All this reminds me of Anna Allison. She was a rising star in 2001. I had dinner with her after she approached me at a conference and begged for a conversation (anyone can talk to me, at any time, if they give me food). At dinner, she mentioned that she was a bug metrics expert. I rolled my eyes and drew a bug metrics graph, daring her to tell me what it meant. What followed was a tour de force of questioning and analysis. She uncovered every trap that I had put into the graph. I told her she should write an article about our conversation and she did!

Tragically, she was on one of the planes that went into the Twin Towers on 9/11, on her way to a consulting gig in LA. This affected me more than I expected it to, because while I didn’t know her well, personally, professionally she was one of the few people I’ve known for whom debate was great fun. The Context-Driven community lost a happy tigress in her. We need more leaders like that. We really couldn’t spare her when she left us, and no one like her has yet stepped up: a non-threatening personality who is a role model for debate. I think that may be why I have high hopes for Lanette. (Also for Meeta Prakash, BTW.)

What happened yesterday?

Yesterday I issued a challenge to new blogger Michael Alexander. He responded promptly and in admirable fashion.

Lanette subsequently did a video blog about why she reacted to my challenge so constructively.

These events inspired me to explain all this. And so, I call upon all testers to challenge me, challenge yourselves, and challenge each other. Let’s blow out the cobwebs. Let’s be testers, not followers.

I need to amplify it because it provides a nice example of at least six useful and important patterns all in one post. This is why I believe Pradeep is one of the leading Indian testers.

Practical advice: “Ask for testability”

His story is all about asking for testability and all the good things that can come from that. It’s rare to see a good example present so vividly. I wanted more details, but the details he gave were enough to carry the point and fire the imagination.

Practical advice: “Try video test scripting”

I have never heard of using videos for scripted testing. Why didn’t I think of that?

Testing as a social process

Notice how many people Pradeep mentions in his post. Notice the conversations, the web of relationships. This aspect of testing is profoundly important, and it’s one that I find Pradeep to excel in. It’s kind of like x-ray vision– the ability to see past the objects of the project to the true bones of it, which is how people think of each other, communicate with, and influence each other. Pradeep’s story is a little bit technical, but it’s mostly social, as I read it.

Experience report

Pradeep’s post is an example of an experience report. Not many of them around. It’s like sighting a rare orchid. He published it with the support of his client, otherwise we’d never have seen it. That’s why there can never be an accurate or profound history written about the craft of testing: almost everything is kept secret. The same dynamic helps preserve bad practice in testing, because that bad practice thrives in the darkness just as roaches do.

Sapient tester blogging

I have referred in the past to a phenomenon I call “sapient tester blogs.” These are introspective, self-critical, exploratory essays written by testers who see testing as a complex cognitive activity and seek to expand and develop their thinking. It’s particularly exciting to see that happening in India, which brings me to the final point…

Leadership in Indian testing

There’s not a lot of good leadership in Indian testing. Someday there will be. It’s beginning to happen. Pradeep’s post is an example of what that looks like.

There must be more than a hundred thousand testers in India. (I wonder if some agency keeps statistics on that?) I would expect to see at least a hundred great tester blogs from India, not six!

“An excellent example of a heuristic is a hammer.”

Ecstasy is your friend: it picks you up at the airport.

Non heuristics that can help an expert solve a problem, without being a guarantee - an abridged list:
* Money
* Time
* Expertise
* Newton-Raphson Algorithm
* Analogies

It was posted anonymously. I generally don’t publish anonymous comments that are argumentative, because opponents who yell things and run away bore me… but this one is helpful. The writer is a little confused, but I bet other people are confused too, in the same way.

He offers a list of things that he claims are non-heuristics (he doesn’t explain why so I guess he thinks they are “obviously” non-heuristics), and suggests that they meet my definition. I think he’s trying to make two points: either my definition of heuristics is wrong because it admits non-heuristics, or it’s trivial because then everything would be a heuristic and therefore the idea conveys no information.

Well, the first point is easily handled. By definition, each thing on his list is heuristic (because he has declared that they help without guaranteeing in some scenario which he seems to have in mind). There is no contradiction, he’s simply mistaken by calling those things non-heuristics.

As to his second point, that’s where the real confusion lies. I think he is mixed-up because he expects heuristics to have some essential nature that identifies them. But what makes something a heuristic is not its essential nature, but rather its relationship to the problem and the problem-solver. We say that anything that may help you solve a problem has “heuristic value” for you with respect to that problem. But if it is infallible (and also if it halts) then we don’t call it a heuristic. We call it an algorithm. For instance, long division is a guaranteed way for dividing one rational number with a finite number of digits by another such number. However, long division is just a heuristic if we were to apply it to splitting the check at a restaurant. It doesn’t guarantee that everyone will feel that they are paying a fair share.

How about instead of heuristics we think about weapons? Would it be absurd of me to suggest that a weapon is any fallible method of attacking your enemy or winning a battle? Someone might reply “But James, are pickles really weapons? Sand? Wind?” The answer is: they can be, depending on your situation. But you don’t need to catalog all possible weapons. Instead you study the art of combat and learn to spot the things that will help you. Myamoto Musashi wrote exactly about this in 1645:

When you attack the enemy, your spirit must go to the extent of pulling the stakes out of a wall and using them as spears and halberds. — Book of Five Rings

We cannot speak intelligibly about heuristics without identifying the problem, the problem-solver, and the dynamics of the situation. You can make a list of anything you want, Mr. Anonymous, but can you answer my next questions, which are: what specific problems are you talking about for which these are heuristics and how specifically are they heuristic? Or if you think they aren’t heuristics, what specific problem do you think they can’t help with and why not?

This is a quick note about three testing heuristics I named this week:

Steeplechase Heuristic (of exploratory boundary testing)

When you are exploring boundaries, think of your data as having to get to the boundary and then having to go other places down the line. Picture it as one big obstacle course with the boundary you are testing right in the middle.

Then consider that very large, long, extreme data that the boundary is designed to stop might founder on some obstacle before it ever gets to the boundary you want to test. In other words, a limit of 1,000 characters on a field might work fine unless you paste 1,000,000 characters in, in which case it may crash the program instantly before the boundary check ever gets a chance to reject the data.

But also look downstream, and consider that extreme data which barely gets by your boundary may get mangled on another boundary down the road. So don’t just stop testing when you see one boundary is handled properly. Take that data all around to the other functions that process it.

Galumphing (style of test execution)

Galumphing means doing something in a deliberately over-elaborate way. I’ve been doing this for a long time in my test execution. I add lots of unnecessary but inert actions that are inexpensive and shouldn’t (in theory) affect the test outcome. The idea is that sometimes– surprise!– they do affect it, and I get a free bug out of it.

An example is how I frequently click on background areas of windows while moving my mouse pointer to the button I intend to push. Clicking on blank space shouldn’t matter, right? Doesn’t hurt, right?

I actually learned the term from the book “Free Play” by Stephen Nachmanovitch, who pointed out that it is justified by the Law of Requisite Variety. But I didn’t connect it with my test execution practice until jogged by a student in my recent Sydney testing class, Ted Morris Dawson.

Creep & Leap (for pattern investigation)

If you think you understand the pattern of how a function works, try performing some tests that just barely violate that pattern (expecting an error or some different behavior), and try some tests that boldly take that behavior to an extreme without violating it. The former I call creeping; the latter is leaping.

The point here is that we are likely to learn a little more from a mildly violating test than from a hugely violating test because the mildly violating test is much more likely to surprise us, and the surprise will be easier to sort out.

Meanwhile, stretching legal input and expectations as far as they can reasonably go also can teach us a lot.

Creep & Leap is useful for investigating boundaries, of course, but works in situations without classic boundaries, too, such as when we creep by trying a different type of data in a function that is supposed to be rejected.

A heuristic is a fallible method of solving a problem or making a decision. Cem Kaner and I came to this definition based on an extensive search of papers and books across fifty years of psychology and engineering. Amazingly, we were not able to find a single coherent definition of heuristic in our research (The dictionaries are not much help, here) . Coleridge, Kant, and Polya have all written about heuristic reasoning. But we needed a definition that would bring the issues into focus.

There are two main issues: something that helps you solve a problem without being a guarantee. This immediately suggests the next issue: that heuristics must be applied sapiently (meaning with skill and care).

An excellent example of a heuristic is a hammer. Do you see how a hammer can help a carpenter solve a problem, but does not itself guarantee the solution? I like this example because it’s so easy to see that that a hammer may be critical to a skilled carpenter while being of little use to an unskilled lout who doesn’t know what to pound or how hard to pound it or when to stop pounding.

Heuristics do not replace skill. They don’t make skill unnecessary. But they can make skilled people more productive and reliable.

How Do You Tell if Consultants/Trainers Understand Heuristics?

I typically hear two reactions from rival testing consultants of other schools of thought (especially the Factory and Quality Control schools) that love “best practice” talk:

1. “Oh yes, heuristics! That’s another name for ‘best practices’, right?”
2. “Oh no, heuristics! That’s just a fancy name for ‘best practices’!”

Obviously both reactions miss the point. Even if these folks rename their ideas of what you should do to call them “heuristics,” they would be leaving out a key idea, which is that skill must rule methods. Talking about methods, focusing on methods, enshrining methods, is only sensible if humans are left in charge. The heuristic nature of engineering is the reason why a “best practice” is an absurdity. Seek not the perfect practice. Seek instead to practice your skills.

My friend Judah Mogilensky once commented that he thought my Heuristic Test Strategy Model was a lot like the Capability Maturity Model that he works with.

“Are people allowed to change or suspend the CMM whenever they see fit?” I asked.

“Oh no,” he replied.

“Then it isn’t being applied responsibly as a heuristic. It’s being treated like a law, or an oath, or something else that places itself above its subjects and binds them.”

So that’s how you tell. Look to see what’s driving things. People, or concepts? Fundamentally, “methodology” can’t control projects. Anytime you seem to see methods in charge, you are actually witnessing a project out of control, or else a project under covert control by shadowy masters.

When someone teaches you a way to solve a problem, check:

• Do they teach you how to tell if it’s working?
• Do they teach you how to tell if it’s going wrong?
• Do they teach you heuristics for stopping?
• Do they teach you heuristics for knowing when to apply it?
• Do they compare it to alternative heuristics?
• Do they show you why it works?
• Do they help you understand when it probably works best?
• Do they help you know how to re-design it, if needed?
• Do they let you own it?
• Do they ask you to practice it?
• Do they tell stories about how it has failed?
• Do they listen to you when you question or challenge it?
• Do they praise you for questioning and challenging it?

A funny thing about that is Stuart once spent 30 minutes trying to convince me that the number “35,000″ was evidence of how great the ISEB certification is, as in “35,000 happy customers can’t be wrong.” Such a concept of “evidence” wouldn’t not pass muster in a freshman course in logic and rhetoric. How does he know that the 35,000 people are happy? How does he know that they are qualified to judge the quality of the certification? How does he explain the easily checked fact that you can pick out any three ISEB or ISTQB certified testers, ask them if they think the certification has made them better testers or indicates that they are better testers, and at least two of them will do the equivalent of rolling their eyes and smirking? (Don’t believe me? I understand. So TRY IT, as I do on a regular basis in my classes)

You might think Stuart is attempting a bold and classic rhetorical move: attempting to control the terms of the debate. The problem he has is that he will lose the debate even faster if he actually engages on the question of evidence. This is because there is plenty of evidence from other fields and the history of thought itself to justify the positions of the Context-Driven School of testing. We are winning the debates because we are better informed and better educated than the Factory Schoolers, for instance, represented by Reid. For instance, Rikard Edgren (who says he’s not in the Context-Driven School, but looks like a duck to me) wrote about applying Grounded Theory to testing. I wonder if Stuart Reid has ever heard of Grounded Theory. He probably has, because I probably mentioned it at least once in the hours of debate that Stuart and I have had. He didn’t respond or react. My impression was that he wasn’t listening.

There’s something far more important than evidence that we need in our industry: engagement. People need to listen to and respond to the arguments and evidence that are already out there.

Here’s one sort of evidence I put in front of Stuart, in a debate. I claimed that my school of testing represents a different paradigm of thinking about testing than his does. After giving him examples of specific words that we define differently and concepts that we arrange differently, it became clear that the deeper problem is that he thought I was pretending to believe things that I don’t believe, just to be difficult. He actually said that to me!

This is the last resort of the determined idealogue: poke your own eyes out so that you don’t risk seeing contrary evidence. Stuart’s case rests on pretending that no one else is making a case! His demand for evidence is meant to give the impression that the evidence is not already sitting in front of him being ignored.

Cem Kaner, Michael Bolton, and I have been marshaling evidence, pointing out the lack of evidence against our ideas, and demonstrating our methods for many years. Next week it will be exactly 23 years since I first became a full-time software tester, and nearly 17 years since the first time I stood up at a conference and pointed out the absurdity of “traditional” testing methods.

BTW, here some of the kinds of evidence I offer when challenged about my work:

• The Sciences of the Artificial, by Herbert Simon (this establishes, based on a body of research for which he won the Nobel Prize in 1978, the heuristic nature of engineering)
• Collaborative Discovery in a Scientific Domain, Takeshi Okada, Herbert Simon, 1997, (this is an experiment that observed the behaviors of scientists attempting to create and perform experiments together in an exploratory way)
• The Processes of Scientific Discovery: The Strategy of Experimentation, Deepak Kulkarni, Herbert Simon, 1988 (this study analyzes the basic exploratory processes of science)

The first item here is a book, the next two are papers published in the journal Cognitive Science. See, if Stuart wants evidence, he has to look beyond the desert that is Computer Science. He needs to get serious about his scholarship. That will require him to find, in his heart, a passion to learn about testing.

(Editors Note: The ISTQB is the organization of cash-grubbing bullies that runs advertisements that say things like “join the elite”– meaning “pay us to give you a ridiculous and unnecessary exam that almost everyone passes so that dimwitted managers will think that you know how to test software.”  If you are ISTQB certified, and you actually know how to test, then you know how silly the exam is.)

I don’t mind getting email. What bugs me here is that the guy, after telling me that the ISTQB is basically a bad organization run by nasty people, then sent me a follow-up letter about how he would sue me if I passed that information on to anyone else.

First he wants to cry on my shoulder, and then he threatens me!

Listen Bernard, don’t threaten me with penalties for telling the truth. The problem I have with you guys is that you lack courage and education. Please don’t expect me to have the same spinning moral compass as you do.

I will not publish verbatim what you sent me. That would violate your copyright. You own the copyright to your own email. But I have made no agreements with you about keeping information secret that you send unsolicited to my inbox on purpose and with full knowledge that I am opposed to the ISTQB and all that it represents. It was obviously intended as a leak. Perhaps instead of threatening me directly, you should tell Rex Black to have his lawyers send a demand letter asking for the return of the memo you leaked to me.

But here’s a better idea: you could have asked me nicely to do what you asked. Perhaps in Belgium they also have this saying “You can catch more flies with honey than with flamey emails.”

If you have made a comment in the last couple of weeks and didn’t see it appear, it maybe lost. Please try again. I did manage to rescue a few.

I’m sorry for the suckiness of modern software…

Actually if anyone knows what causes Wordpress to put something in the spam folder, I’d love to hear from you.

– James

I recalled that Steve McQueen played my kind of expert, in the movie The Towering Inferno. So, I found a clip and added subtitles to turn it into a testing situation. I hope you enjoy it.

Pay attention to how McQueen throws questions at a rapid fire pace. He knows what he needs to discover, he organizes people around him, and he forms specific plans, quickly, to get the job done…

(CNN) — Early evidence points to driver error as the reason a 2005 Prius sped into a stone wall on March 9, according to federal investigators.

“Information retrieved from the vehicle’s onboard computer systems indicated there was no application of the brakes and the throttle was fully open,” according to a statement from the National Highway Traffic Safety Administration.

The statement suggests the driver may have been stepping on the accelerator, instead of the brake, as she told police.

Note to CNN: Information retrieved from an onboard computer is not reliable in cases where the computer itself is a suspect in the crime. No one is claiming that ghosts are causing the cars to go out of control. And there is no evidence yet that a mechanical failure is the culprit, despite the NHTSA looking hard for that evidence. Alcohol is also not a factor in these specific cases (or else we wouldn’t even be talking about them). That leaves two big possibilities: sober experienced people are suddenly forgetting how to drive OR something is wrong with the software or electronics that controls the cars.

If the software that reads the control inputs has the right kind of fault in it, it may occasionally lose the ability to read and react to control inputs. It is easy for a computer programmer to imagine a situation where the part of the system that records the control settings is working fine, while the part that acts on them is failing. That would be consistent with the facts of this case. Moreover, the problem may be transient, leaving no evidence that it happened.

We don’t know what happened. We may never know. But there is no reason to assume that the computer is infallible. Computers are designed by fallible people.

I also don’t know what the real story is with this Sikes fellow who was in the car that he claims went out of control. Maybe he’s a gold digger out to exploit Toyota’s bad fortune.

Finally, I don’t know much about the design of a Prius, except that my understanding is that there are no physical cable linkages. It’s software driven. When you press the brakes you are essentially double clicking on the “brake” icon with your foot mouse, hoping that the operating system agrees to apply the real brakes. A Prius is basically a video game console connected to a car. You drive the console, not the car.

What I’m trying to do is use this situation to teach testing. So, I want to break down a news story I found on cnn.com and show you how a tester would think about it:

Toyota takes aim at California runaway Prius story
by Peter Valdes-Dapena, senior writerMarch 15, 2010: 8:24 PM ET

NEW YORK (CNNMoney.com) — Toyota challenged a California driver’s story of an out-of-control Prius at a press conference Monday afternoon.

Toyota held a press conference about this. That is interesting. Of course that means Toyota’s marketing people are actively involved in this investigation, pacing and fretting for good news to send out. This does not create a conducive atmosphere for an investigation.

Public relations people want to rush out with good news, but keep bad news longer to study it and make very sure it’s really correct. This creates a sort of chromatic distortion of the truth in the near term. We must beware of that. You might say that it’s likely to be “green shifted” truth.

Company executives detailed preliminary findings of a joint investigation conducted by Toyota and the National Highway Traffic Safety Administration into the incident.

I like hearing that the NHTSA is involved in the investigation. Are they leading it, I wonder? You know there are several ongoing investigations into Toyota vehicles by the NHTSA (see their website) including one involving the momentary loss of braking on uneven surfaces (which my father has experienced several times on the dirt road going to his house.)

I feel there’s more credibility if the NHTSA is involved, but the press conference seems to have been a Toyota thing, not an NHTSA thing.

Prius owner Jim Sikes made national headlines last week with claims that his car’s accelerator got stuck as he sped up to pass a car while traveling on California’s I-8 highway outside of San Diego, and that he was unable to stop the car.

“As I was going, I was trying the brakes … and it just kept speeding up,” he said.

Reports from non-technical, non-expert users always must be taken with skepticism, even leaving aside the possibility that the guy is just telling lies. Perhaps he suffers from Munchhausen syndrome. You never know.

But let’s say he’s not lying, just for a moment. The phrase “it just kept speeding up” suggests that the brakes were completely inoperative. I wouldn’t use that phrase, as a driver, if the brakes had engaged and were fighting the motor.

Someone needs to sit the guy down (I assume they’ve done this) and walk through the whole incident moment by moment. Several times. Get him to clarify this.

Inconsistencies alone should not worry us too much. It happened quickly, it was a traumatic event, and his personal account may not be reliable just for that reason. But we still need to wring every bit of information we can from his memory.

Sikes story is at odds with the findings of the investigation, according to Toyota and to a draft congressional memo obtained by CNN.

“While a final report is not yet complete, there are strong indications that the driver’s account of the event is inconsistent with the findings of the preliminary analysis,” Toyota said in a prepared statement.

Sikes said he called 9-1-1 for help as he was traveling in excess of 90 mph on a winding, hilly portion of the interstate. He said dispatchers tried to talk him through ways to stop the car, but nothing helped.

I’d like to hear that 911 call in its entirety.

Eventually, a California Highway Patrol officer was able to catch up to Sikes and used the patrol car’s public address system to instruct Sikes to apply the brakes and the emergency brake at the same time. That tactic worked, and he was able to stop the car.

I’d like to know if Sikes had tried the emergency brake alone, before this. Had he tried both at once before this.

I can understand if he did not try both at once, because a normal driver would think if they don’t work individually, why would they suddenly work together? But  in a drive-by-wire system, everything is mediated by software, and software can get into strange states. It’s technically plausible that only with both brake controls activated the software could be bounced out of whatever strange trance it got into.

However, because driving a hybrid car like the Prius with both the gas pedal and the brakes simultaneously depressed would cause serious damage to the car’s electric motor and, possibly other systems, Toyota says the Prius is designed to prevent that from happening.

Of course it’s designed prevent that. But here’s a testing lesson: designing for prevention is not the same as preventing, because your design may have a bug in it.

All Toyota can say is that it was their intent to design the system to prevent that, and to the best of their knowledge that is how the system works… except in this case it didn’t work– unless the guy is simply lying or insane.

If the brake is pressed at the same time as the gas pedal, power to the engine will be reduced just as if the gas pedal had been released, the automaker said.

Unless, of course, there’s a malfunction of the system, which is exactly the issue under consideration.

During driving tests on Sikes’ Prius and on an identical 2008 Prius, the system operated as expected, according the report, preventing the car from pushing forward while braking.

“The system operating as expected” is not actually possible to determine, because they can’t see inside all the software and hardware to detect that every bit and electron is in the right place.

What they can say is that they detected no problems. Problems may be there, they just did not detect them.

If the visible, detectable problems we want to see are triggered by a transient event, such as a specific combination of foot presses, or perhaps there’s a two microsecond window of opportunity for two software events to happen simultaneously (such as have dogged the Mars rover missions ), then of course driving it around a parking lot is probably not going to reproduce the problem.

It is also possible that part of the problem involves a piece of physical equipment that was lodged or worn in a certain way at a specific temperature, and that condition no longer exists on the car in question.

When we try to reproduce problems, we often have to guess at the causes, and we may guess wrong.

If I were Toyota, I would treat this like an epidemiology problem. You interview people and make a list of absolutely everything that was going on. Did they have a cell phone? What kind? Where was it in the car? Where they using the cup holder? What drink was in the cup? Hot or cold? Was the air conditioner on or off? What was the setting?

Then you put all the data into a database and mine it for patterns.

Investigators are extremely meticulous when taking apart a car in a case like this, said Ed Higgins, a Michigan personal injury attorney who has been involved in automobile defect cases. They are aware their work will be subject to intense scrutiny, so they measure and document everything, he said.

That kind of care takes a lot of time. But it hasn’t been very long since the incident occurred. Have they also taken the software apart? Have they comprehensively reviewed the code? I seriously doubt that.

“I would think that any mechanical defect that would have allowed something to happen that otherwise could not have happened would have stood out like a sore thumb,” he said.

Unless it’s a transient interaction between a mechanical defect and an invisible state within the software.

The car also did not show damage consistent with the engine having been running at full throttle while the brakes were on, according to the report.

That suggests the brakes weren’t on, but not that Sikes wasn’t pushing on the brakes.

“Toyota engineers believe that it would be extremely difficult for the Prius to be driven at a continuous high speed with more than light brake-pedal pressure, and that the assertion that the vehicle could not be stopped with the brakes is fundamentally inconsistent with basic vehicle design and the investigation observations,” Toyota said in a statement.

Again, this all assumes normal circumstances and no transient failures. For the purposes of investigation, that belief is irrelevant.

It is already fundamentally inconsistent with the design of the product that ANY failure could occur. We’ve crossed that bridge, guys.

Remember, when flight 427 crashed, Boeing maintained for years that their rudder mechanism could not possibly have failed– until a new form of failure was discovered (”thermal shock”) and the specific failure reproduced in that very rudder assembly.

The car’s front brakes showed significant wear and overheating, Toyota said. That kind of wear and heat would be consistent with the brakes being lightly applied over a long period of time, executives said at the press event.

Data from on-board computers indicated that Sikes had applied the brakes, to some degree, at least 250 times during the 23 mile incident, Toyota executives said, and that the brakes worked normally each time.

Ooh, I love log files. I wonder what other patterns they can mine from that log file?

If the computers indicate that Sikes had applied the brakes, that shows they were getting some kind of signal from the brake mechanism, but not necessarily the correct signal. Therefore saying “the brakes worked normally each time” is completely unwarranted. Part of the system may have been working normally while another part was going haywire. There’s not way to tell after the fact, because “working normally” is not a detectable condition that gets logged in computer.

Every time you experience a problem in your software, your software, on some level, thinks it is doing the right thing. Software doesn’t “know” it’s misbehaving. It just does what it is told.

Edmunds has independently tested Prius cars similar to Sikes’ and confirmed that the engine would stay engaged if the brakes were only pressed lightly, but not hard enough to actually stop or slow the car, said Dan Edmunds, head of auto testing for the automotive Web site Edmunds.com.

He says “would”, but he should say “would, assuming that there is nothing wrong with the car that would cause it not to”

“If you’re just riding the brakes, it will ride the brakes,” he said.

“These findings certainly raise new questions surrounding the veracity of the sequence of events that has been reported by Mr. Sikes,” said Kurt Bardella, spokesman for Rep. Darrell Issa, R-Calif., and ranking member of the committee.

Sikes’ attorney, John Gomez, denied that the report proves his client was wrong about what happened to his car.

“The notion that they weren’t able to replicate it in this particular case tells us nothing,” he said. “They haven’t been able to replicate a single one of these.”

That’s right. Also, learn the phrase “transient failure mode” and press that point. There are plenty of examples in space missions and airliners of such failures.

Sikes has no plans to sue Toyota, Gomez said.

Gomez is also representing the family of Mark Saylor, a California Highway Patrolman who was killed, along with members of his family in a Lexus sedan that accelerated out of control. A preliminary investigation has found that the accelerator pedal in that car probably became trapped on an all-weather floor mat that had been incorrectly installed in the vehicle.

Toyota has issued a recall for several models, including Sikes’ Prius, to address possible floor mat entrapment. Sikes’ floor mat was not interfering with the accelerator, investigators found, and there were no signs the pedal had become stuck in any way, according to the report.

The investigators findings “suggest that there should be further examination of Mr. Sikes account of the events of March 8,” Toyota said in its statement.

Toyota spokesman Mike Michels also took issue with media coverage of the Sikes incident. Journalists sensationalized an admittedly dramatic event, he said, but the public would have been better served had reporters waited for all the facts.

“We need to let investigations take their course,” he said.

Yes indeed. And this investigation has not done that. Be careful what you wish for, Mr. Michels.