A press release by Toyota recently stated:

Toyota’s electronic systems have multiple fail-safe mechanisms to shut off or reduce engine power in the event of a system failure. Extensive testing of this system by Toyota has not found any sign of a malfunction that could lead to unintended acceleration.

Here are some notes for the lawyers suing Toyota. Here is what your testing experts should be telling you:

• Whoever wrote this, even if he is being perfectly honest, is not in a position to know the status of the testing of Toyota’s acceleration, braking, or fault handling systems. The press release was certainly not written by the lead tester on the project. Toyota would be crazy to let the lead tester anywhere near a keyboard or a microphone.
• Complete testing of complex hardware/software systems is not possible. But it is possible to do a thorough and responsible job of testing, in conjunction with hazard analysis, risk mitigation, and post-market surveillance. It is also quite expensive, difficult, and time consuming. So it is normal for management in large companies to put terrible pressure of the technical staff to cut corners. The more management levels between the testers and the CEO, the more likely this is to occur.
• “Extensive testing” has no fixed meaning. To management, and to anyone not versed in testing, ALL testing LOOKS extensive. This is because testing bores the hell out of most people, and even a little of it seems like a lot. You need to find out exactly what the testing was. Look at the production-time testing but focus on the design-time testing. That’s where you’ll be most likely to find the trouble.
• Even if testing is extensive in general, you need to find out the design history of the software and hardware, because the testing that was done may have been limited to older versions of the product. Inadequate retesting is a common problem in the industry.
• If Toyota is found to have used an automated “regression suite” of tests, then you need to look for the problem of inadequate sampling. What happens is that the tests are only covering a tiny fraction of the operational space of the product (a fraction of the states it can be in), and then they just run those over and over. It looks like a lot of testing, but it’s really just the same test again and again. Excellent testing requires active inquiry at all times, not just recycling old actions.
• If Toyota is found not to have used test automation at all, look for a different kind of sampling problem: limited human resources not being able to retest very extensively.
• Most testers are not very ambitious and not well trained in testing. No university teaches a comprehensive testing curriculum. Testing is an intellectually demanding craft. In some respects it is an art. Examine the training and background of the testing staff.
• Examine the culture of testing, too. If the corporate environment is one in which initiative is discouraged or all actions are expected to be explicitly justified (especially using metrics such as test case counts, pass/fail rates, cyclomatic complexity, or anything numerical), then testing will suffer. During discovery, subpoena the actual test reports and test documentation and evaluate that.
• Any argument Toyota makes about extensiveness of testing that is based on numbers can be easily refuted. Numbers are a smoke-screen.
• Examine the internal defect tracking systems and specifically look to see how intermittent bugs were handled. A lack of intermittent bug reports certainly would indicate something fishy going on.
• Examine how the design team handled reports from the field of unintended acceleration. Were they systematically reviewed and researched?
• Depositions of the testers will be critical (especially testers who left the company). It is typical in large organizations for testers to feel intimidated into silence on critical quality matters. It is typical for them to be cut off from the development team. You want to specifically look for the “normalization of risk” problem that was identified in both the Columbia and Challenger shuttle disasters.
• If the depositions or documentation show that no one raised any concerns about the acceleration or braking systems, that is a potential smoking gun. What you expect in a healthy organization is a lot of concerns being raised and then dealt with forthrightly.
• Find out what specific organizational mechanisms were used for “bug triage”, which is the process of examining problems reported and decided what to do about them. If there was no triage process, that is either a lie or a gross form of negligence.
• If Toyota claims to have used “proofs of correctness” in their development of the software controllers, that means nothing. First, obviously they would have to have correctly used proofs of correctness. But secondly, proofs of correctness are simply the modern Maginot line of software safety: defects drive right around them. Imagine that the makers of the Titanic provided “proof” that water cannot penetrate steel plates, and therefore the Titanic cannot sink. Yes steel isn’t porous, but so what? It’s the same with proofs of correctness. They rely on confusing a very specific kind of correctness with the general notion of “things done right.”
• The anecdotal evidence surrounding unintended acceleration is that it does not only involve acceleration, but also a failure of braking. Furthermore, it’s a very rare occurrence, therefore it’s probably a combination of factors that work together to cause the problem. It’s not surprising at ALL that internal testing under controlled conditions would not reproduce the problem. Look at the history of the crash of US Air flight427, which for years went unsolved until the transient mechanism of thermal shock was discovered.
• You need to get hold of their code and have it independently inspected. Look at the comments in the code, and examine any associated design documentation.
• Look at how the engineering team was constituted. Were there dedicated full-time testers? Were they co-located with the development team or stuffed off in another location? How often did the testers and developers speak?
• What were the change control and configuration management processes? How was the code and design modified over time? Were components of it outsourced? Is it possible that no one was responsible for testing all the systems as a whole?
• What about testability? Was the system designed with testing in mind. Because, if it wasn’t, the expense and difficulty of comprehensive testing would have been much much higher. Ask if simulators, log files, or any other testability interfaces were used.
• How did their testing process relate to applicable standards? Was the technical team aware of any such standards?
• In medical device development, manufacturers are required to do “single-fault condition” testing, where specific individual faults are introduced into the product, and then the product is tested. Did Toyota do this?
• What specific test techniques and tools did Toyota employ? Compare that to the corpus of commonly known techniques.

The best thing would be to reproduce the problem in an unmodified Toyota vehicle, of course. In order to do that, you not only need an automotive engineer and an electrical engineer and a software engineer, you need someone who thinks like a tester.

The unfortunate fact of technological progress is that companies are gleefully plunging ahead with technologies that they can’t possibly understand or fully control. They hope they understand them, of course, but only a few people in the whole company are even competent to decide if that understanding is adequate for the task at hand. Look at the crash of Swiss Air flight 111, for instance: a modern aircraft brought down by its onboard entertainment system, killing all aboard. The pilots had no idea it was even possible for an electrical fire to occur in the entertainment system. Nothing on their checklists warned them of it, and they had no way in the cockpit to disable it even if they’d had the notion to. This was a failure of design; a failure of imagination.

Toyota’s future depends on how they take seriously the possibility of novel, multivariate failure modes, and aggressively update their ideas of safe design and good testing. Sue them. Sue their pants off. This is how they will take these problems seriously. Let’s hope other companies learn from no-pants Toyota.

My son turned 16 a couple of weeks ago, and the event prompted me to take a fresh look at him. This is hard to do, in life. You put someone in a category, and it’s all too easy to keep him there. I think my own mother still thinks I’m 14 years-old (the age at which I moved away from home). She reacted to me then just as she reacts to me now, not quite believing that I’m an adult (I’m afraid I don’t get along much with my mother.) But with my father, I’ve become good friends. He saw the mistakes I made and phases I went through, years ago, but recognized that I grew through them.

So, my son is becoming a man. There’s more depth to him, this year. He’s a little more reliable. He seems to have plans for the future. His confidence is improving. I must take this fellow more seriously.

And now for the segue: The same is true about Indian software testing culture. It’s still, on the whole, pretty unremarkable, but there’s a wonderful bright creativity bursting out. It demands our attention. Events are happening quickly, now.

Today I’ve added another Indian tester’s blog to my blogroll. Dhanasekar S’s blog is part of a bona fide trend: sapient testing blogs. These are blogs by people who write exploratory essays (note that the term “exploratory essay” predates exploratory testing, but means something quite similar) that analyze the progress of their thinking as they test. They try to open the black boxes of their minds for us to see inside. They don’t just recycle folklore about tools or processes or rules or definitions, as most Indian testing blogs have traditionally done.

Dhanasekar’s entries include “I can’t Nike because I Rebok” which whimsically declares sapience, and “Unbelievable, Up to 80% off*” which skewers claims made by automation sales people.

This is something very fresh in Indian testing culture. I believe it’s a dam that is in the process of bursting. I don’t know how many testers there are are in India. It’s got to be at least 100,000. If just one of two hundred of them can write and have ambition, they will make a tremendous impact on the direction and development of the testing craft.

I believe Pradeep Soundararajan’s blog was the first of these self-aware testing blogs. The next one I knew about was Sajjadul Hakim’s blog (yeah, I know, he’s in Bangladesh, but look at a map. It’s right nearby). For a long time, those were the only two. In recent months, perhaps spurred by the Weekend Testers, more have cropped up. A canonical example is Parimala Shankaraiah and her account of the 30-minute challenge. What a spirit of inquiry! Santosh Shukla blogged about a challenge I gave him and how he reacted to it. I came across Sharath Byregowda, while googling for more. He’s at Mindtree, where I taught a class, some years ago, hoping to plant seeds. Nandagopal has just started a blog, I hope he sticks with it. He’s an example of a relative fresher using blogging to help focus his thoughts.

There are sapient testing bloggers around the world. I’m singling out India because there are so many testers there and this has been so out of character for them up to now.

The sapient bloggers are re-examining patterns that have surrounded testers since the beginning, but that most of us take for granted. This is also a secret to my success. Unlike most people, I don’t think it’s a waste of time to ask myself questions like “How do I know how to open a door in a building I’ve never before been to?” or “How do I decide that a pattern on my screen represents a button and not background art?” To many people around me, it is enough to trust in the magic of human inductive reasoning. They are satisfied that a little elf inside their heads just knows how to see things and recognize them and test. But I am not satisfied with that. I don’t consider thoughtlessness to be pragmatic. Neither do the members of this new Indian wave.

First there was Pradeep. Now there’s maybe a dozen others. I predict, by two year from now, there will be a hundred of them, and someone in India will have organized the first Sapient Testers Conference. I hope I am able to attend.

P.S. I spoke to Aparna Sharma, who runs the testing division at Infosys (the division is called IVS, but I think that’s a fancy was of saying testing). I expect to be blogging soon about what she told me about how they develop testing expertise at Infosys. Her answers were impressive, but I’d like to hear from actual Infosys testers in order to get a sense of the ground truth.

Richard drove up to the hangar just as I was checking the oil on the Husky, his prized baby float plane. Nuts. He was right on time. I was late. I’m supposed to have the plane ready to go when he arrives.

“Hey Dad, looks like a good day for flying. I’m just in the middle of the pre-flight.”

“Where are we going today?” He asked.

“I haven’t been up for a few months, so I figured just a sightseeing tour around the islands and then some pattern work at Friday Harbor.” I hate pattern work: landing and taking off while talking on the radio to the other pilots. That’s exactly why I need to do it, though. I must get over my nerves; must become a safe pilot. It’s a lesson from testing: focus on the risk.

“How much fuel do we need for that?”

“There’s about 18 or 20 gallons on board. That’s actually enough, but I figure it would be better to bring it to 35, just in case.”

“How much will you put in each tank, then?”

“7 gallons.”

“7 plus 7 plus 18 doesn’t add up to 35. Decide on the fuel you want and get that. If you’re going to fudge, fudge toward having more fuel. What are the four most useless things in the world?”

Oh I know this… “Um, altitude above you… runway behind you… gas in the gas truck, and… Um–”

“–and a tenth of a second ago” he finished. “But you remembered the important one. Gas. We don’t want that terrible feeling when they close the airport for 30 minutes to clean something up on the runway, and we don’t have the fuel to divert.”

I could have quibbled with him. What we would actually do in that situation is land 10 minutes away at Friday Harbor airport, or heck, anywhere, because we’re a float plane in the midst of an archipelago. But that’s not the point. The point was the habit of precision; of being conscious and calculated about the risks I take, wherever reasonable. Again this relates to testing. When I’m testing, the habit of precision comes out in considering and controlling the states of my test platforms and test data, and of knowing what to look for in test results.

Dad called the flight center for a briefing. He already knew what they’d say, since he always checked the weather before he left home, but Richard Bach is an especially fastidious pilot. He’s not exactly “by the book.” It’s more that he prides himself on safety. Nothing surprises him without itself being surprised at how prepared he is for it. Yes he knew you were coming; here’s your cake.

The Tester’s Attitude in the Air

My father’s philosophy dovetails perfectly with the tester’s mindset. We expect things to go wrong. We know we will be surprised at times, but we anticipate the kinds of surprises that might occur. That means we perform checks that might seem unnecessary at times. And we keep our eyes open.

I was almost done with the walkaround when he got off the phone.

“Three knots at zero six zero over at Friday,” he announced.

I paused to visualize, then tried to sound authoritative. “That’s a crosswind favoring runway three four.”

“Yes. We have the same conditions here.”

Cool. I got it right. I’m supposed to pretend to be the pilot. Officially, Dad is the pilot-in-command, but I do everything he would do, while he supervises and is ready to take over in case there’s a problem. While I’m doing the preflight, he’s doing it too, but not telling me anything– unless I miss something important. Each time we fly, I’m determined to find a problem with the aircraft that he hasn’t noticed, first. I haven’t yet succeeded.

“Dad, what’s this rust colored streak coming out of this bolt?” Yay, I found something! “There’s one of each side of the elevator.”

“Just a little bit of rust.” He smiled and materialized a can of WD-40 and blasted the bolts with it. This airplane is pristine, so even a little blemish of rust really stands out.

“Were you flying recently?”

“Yeah, I went out last week and splashed around at Lake Watcom.”

“That explains the streaks. Water spray on the tail. Did you pump out the floats afterward?”

“No, but I doubt there’s more than a pint of water in there.”

“Let’s see about that.” I retrieved the hand pump while he popped out the drain plugs. He was right again, I couldn’t suck out more than a cup of water from the floats, total, from all the compartments.

But there was something odd about the last one.

“This water is PINK, Dad!”

Now he was not smiling.

“Unless you landed at the rainbow lake on Unicorn Planet, there may be a hydraulic leak in there.”

He put his fingers in the residue and sniffed it like a bush guide on the trail of a white tiger. “Yeah, that’s what it is. Let’s pop the hatch and take a look.”

Testing With Open Expectations

This is an example of why good testing is inherently an open investigation. Yes, I had definite ideas of what I was testing for: leaky floats. Yes, I had a specific method in mind for performing that test. Had I not a specific method, I would have developed one on the fly. That’s the testing discipline. My oracles were judging the amount of water I was able to pump out of the floats compared to other occasions, and I also tasted the water a couple of times to detect if it was salty. It shouldn’t be, because it had been several flights since we had landed in salt water, but I check just in case there was a previously undetected leak from before. If salt water gets in there, we could have a serious corrosion problem.

I had no conscious intent to check the color of the water. But in testing we take every anomaly seriously. We take data in and ask ourselves, does this make sense? In that way, we are like secret service agents, trained to deal with known threats and to have a good chance to discern fresh and unusual threats, too.

The question “Can I explain everything that I see?” is a good heuristic to keep in mind while testing.

But if I were to have automated my float pump tests, I would never have found this problem. Because unlike a human, a program can’t look at something and just try to explain it. It must be told exactly what to look for.

I got an email, today…

Geoff confirmed the hydraulic leak at the connection in the left float, and will be sealing it, probably tomorrow.  He’ll move the Husky to the big hangar to do the work. Nice, that you decided to pump the floats!

Dad

I’m on a new project lately, working with a team at QualiTest. We’re testing a class III medical device. This is an exciting project, because for the first time I am aware of, formalized exploratory testing will be used to do such a validation. We will not rely on masses of procedural test scripts. I’ve been called in on this project because I created the first published formalized ET process in 1999 (for Microsoft), and created, with my brother Jon, session-based test management, which is basically a general form of that Microsoft process.

The QualiTest team consists of senior testers hand-picked for this job, who have regulatory testing backgrounds and an enthusiasm to use their brains while they test. On top of testing well, we have to document our testing well, and trace our testing to requirements. Automatic logging is one of the tools that will help us do that.

I am amazed at how crazy nuts some people get over documentation– how they sweat and shiver if they don’t have a script to cling to– and yet they don’t spare a thought for logging. Logging is great for testers, programmers, and technical support. Logging is automatic documentation. Sing the praises of logging.

I’m talking about function-level logging built into the products we test.

If you test a web app, you already have this (the web server and application logs, plus the use of a proxy to log locally, if you want) or would have it with a little tweak here and there by the programmer. For desktop apps, the programmer has to build it in. Here’s why he should do that right away:

1. Instead of following a script written weeks or months ago by some over-literal, function-besotted and data-blind intern, the tester can think, explore, play, and maintain the thread of inquiry without worrying that you won’t know what you tested, later on.
2. Instead of remembering what you tested, the product tells you how you tested it. Process the log with a simple Perl script, and you can potentially have an automatically generated test report.
3. Instead of just wondering how you made that crazy bug happen, the developer can consult the log.
4. Instead of asking the customer what he was doing moments before the crash, he asks for the log.

If logging is built into the base classes of the product, very little coding is involved.

This idea first occurred to me in 1993, after hearing from John Musa about how his telecom systems would “phone home” with data about how they were being used, but I couldn’t get a programmer to put logging into anything I tested until I was at SmartPatents in 1997. Since then I’ve helped several projects, including a couple of medical device projects, get going with it.

On this most recent project I was asked to create requirements to specify the logging. Here is the generic version of what I came up with:

1. Each significant action that the user takes shall be logged. (pressing buttons, touching screen objects, turning knobs, startup and shutdown, etc.) This provides critical information needed to demonstrate test coverage during validation, and improves our ability to meet and exceed regulatory requirements.

2. The results of any diagnostic self-tests or assert failures shall be logged.

3. Any function should be logged, regardless of user action, that causes a change to data, screen display, system configuration, modes or settings, communicates with other equipment, or produces an error or information message.

4. Everything that could be interesting and useful for testing, support, and system debugging should be logged UNLESS the event occurs so frequently (many times a second) that it poses a performance or reliability risk.

5. Each log event shall include at least the following information:
- Time stamp: For instantaneous events, time stamp (millisecond resolution). For events over time log the start and stop times by logging it as two separate events (e.g. “Event START”, “Event END”). Events that set a persistent mode or state can be logged as one event (”high security mode ON”) but the state of any such modes shall be automatically logged at startup and shutdown so that a complete record of that setting can be maintained over time.
- Event type ID: always unique to event type; IDs not re-used if an event is retired and a new event is created.
- Event type description: short, unique human readable label
- Event information: any data associated with the event that may be useful for customer service or assessing test coverage, this data may be formatted in ways specific to that event type.

6. At startup and shutdown, the current settings, modes, and confuguration shall be recorded to the log.

7. Any errors shall be recorded to the log, including the actual text of the error message.

8. Every type of loggable event shall be stored in one table in the source code or in a data structure accessible on the system itself, such as a header file, enum, array or resource file. This facilitates providing the validation and customer service teams with a complete list of all possible events.

9. The log format shall be in text form, structured and delimited consistently such that it can be parsed automatically by a third party tool. The data for each event should be on one line, or else be marked with standard start and end markers.

10. The log format should be structured and delimited such that it is reasonably human readable (such as tab delimited).

11. The level of detail included in the log file should be configurable in terms of preset levels: 1- error and service events only, 2- Functional events, error events, service events, 3- All events including diagnostic information messages about internal states and parameters.

12. The log should behave as a ring buffer with a maximum of X events (where X is configurable within some limit that would not be exceeded in 7 days of heaviest anticipated use). If the size of the log exceeds available space, the oldest events shall be discarded first.

13. When the log is exported, it should have a header that identifies the software version (and serial number of the HW, if applicable) and current configuration.

My brother and I are experimenting with short podcasts. Here are the first two 15-minute segments:

• Testers say the darndest things. One issue in coaching testers is getting them to speak carefully about evidence and about what they can and can’t do. For instance, we talk about how we react when someone tells us that “a tester ensures the quality of the product.” Jon’s strategy is to point out the incompleteness of testing. Testing can’t be perfect, therefore testers can’t ensure quality. My strategy is to focus on the control aspect: testers aren’t in control of the project, and therefore can’t ensure anything. We think that part of the reason testers make over-the-top statements is that it’s comforting to cling to simple fairy stories about testing.

• The trap of not asking questions. Jon and I notice that testers presented with challenges and problems often jump right into test execution instead of asking questions. We question this phenomenon.

One of the things I like to do is give testing lessons over Skype. I offer this as a service for pay, and I also do it for free under certain conditions.

Here’s how it works:

1. You contact me over Skype. (ID: SATISFICE)

2. If I’m not charging you than it’s on an “as available” basis: How do you know I’m available? Well, if I’m online, then Skype me and ask if I’m available. For free lessons, I’d rather not schedule anything in advance.

3. My Motivation: I am motivated by people who entertain me with their passion for learning. Besides that, I want learn how to coach better. Finally, I’m developing material for another book. But here’s my business reason: I want to publish the edited transcripts of my coaching sessions. So, I need from you permission to include the session in my writing (with your name attached or removed as you wish).

4. The process is Socratic dialog: That means I ask you questions and pose problems, then you develop your answers. I also instruct directly as needed. My goal is to help you develop into a tester worthy of respect according to the standards of the Context-Driven School of Testing.

5. We work on things that matter to you: If you have a goal to learn a specific kind of thing, we work on that. Otherwise, I’m happy to walk you through my own syllabus of testing.

6. I may require you to do homework as a condition of further coaching. I may ask for permission to publish your homework as a training example.

7. I may want to watch you test over Skype screen sharing.

8. If you are a beginner, I may require you to work with another coach first (one of my students or colleagues) as a condition of working with me.

9. Our sessions are not secret unless you want them to be. You are free to tell others that you are working with me.

10. If you ask me a specific question, I will probably ask you to tell me your answer, first. I will expect you to have googled around and thought about it some before we talk.

You should bear in mind that I’m a demanding tutor. I want to spread clarity and excellence in software testing. Don’t approach me unless you have an ambition to be excellent in your craft.

My offer is open to anyone in the world, as long as I believe I can help you.

I recently got back from the Oredev developer’s conference, in Malmö, Sweden. I enjoyed it, though I wish I’d had more time to actually go to the various developer-oriented talks. The closest I got to that was being on this panel:

This is probably the most fun panel I’ve been on. I loved the enthusiasm and wit of my fellow panelists.

(Editor’s note: oops, I originally  put the beginning of another blog post down here. I’m taking it out to put it into its own entry.)

My work is greatly influenced by fellow students of the craft. My friends are critical to my education.They help me focus my attention and they stir up my ideas.

Here are some substantive examples from just the last two days in my life as a student of software testing:

• David Gilbert Skyped me with a new version of his TestExplorer tool. I’m using it to develop examples for my next book, tentatively titled “Real Exploratory Software Testing” to contrast with the shallow knock-off of an ET book out there that has a similar name.
• Michael Bolton roused me out of bed to talk about a change he wants to make to the Rapid Testing Class.
• Pradeep Soundararajan Skyped me to talk about Weekend Testers.
• Ajay Balamurugadas posed me a math problem, and I wrote him about the solution.
• Parimala Shankaraiah emailed me asking for a testing challenge to pique the skills of the Weekend Testers.
• Liz Marley blogged wonderfully about learning challenge I gave her, and I remembered that I have to follow up on my part of that challenge. She also offered a good example of using emotions and a test oracle.
  
• Karen Johnson Skyped me about the protocol of collegial critique between us. She reminded me that I am suppose to critique her work. I promised to.
  
• Matt Heusser Skyped me about writing an article for him. We’re doing it on his wiki. First time I’ve written an article that way.
• Sohail Hasware Skyped me to complain about the stupid testing curriculum he’s forced to go through to get his IT certificate. I once again tried and failed to get him to quit that program and stop wasting his precious time and life consorting with fools.
• Buddy Parker, a 14 year-old home schooled kid also living on Orcas Island, emailed me about how to learn HTML, so I responded with links and some examples.
• Adam Roy, a 7 year-old kid who likes algebra, son of author Jennifer Roy, conversed with me about learning Python.
  
• Ravisuriya continued an email conversation with me about context-driven testing and priority-severity distinctions.
• Adrian Segar and I conversed about the etymology of the term “Peer Conference.” Apparently he’s written a book about it.
• Rodney Thompson emailed me to sound me out about a new term: “situational testing.” I’m dubious about it, but we’re still talking.
• Jonathan Bach alerted me to a new article attacking “best practices”, so I commented on that. He also wants a new article from me.
• Elisabeth Hendrickson offered an apt and testerly reply to my puzzle “guess the next term in the series: tom, dick, ?” She wrote “Bill, if the sequence is from Thomas Richard Williams, pioneer of stereoscopy.” I was thinking “harry”, based on a common cultural pattern, but she demonstrates that other answers are plausible, as a good tester should.
• Henrik Andersson Skyped me about how he might be coming to the U.S.A. at the special invitation of the Association for Software Testing. Henrik is a rising star as a Context-Driven guy in Sweden.
  
• Jerry Gao, a tester from China, wrote to ask how to advocate for exploratory testing over there. I’m replying to him, shortly.
  

These are all happy examples. I’m not leaving anything out. I just don’t happen to be in a flame war with anyone, at the moment.

And thank you Kathy…

I want to mention the most recent useful criticism I’ve received from a colleague. It was from Kathy Iberle, at the PNSQC conference. I had just delivered a presentation I felt good about, on the “myths of rigor.” Kathy said she was expecting me to provide more details and analysis about the alternative approach to rigorous engineering that I had proposed. She said, “I expect to hear something new when I hear a talk from you.” Well, actually, I thought I had presented something new, but as I thought over it again, I realized that I’d left some big fat loose ends in the talk. So, just as I was about to argue with her, agreement stepped in and threw a blanket over my cage.

Kathy is an interesting example of someone who has gained her reputation in my community entirely without– I want to say trying, but that’s not right– Kathy doesn’t push. It’s actually frustrating to me, who does push. I like turning over bee hives to get the honey, but with colleagues like Kathy, I have to wait patiently for the bees to fill tiny thimble jars and fly by with them. What honey, though!

The main reason I know about Kathy is from seeing her at peer conferences. That’s the single best mechanism I can recommend for building mass collegiality.

(BTW, my Skype ID is “satisfice”)

There’s something important going on in India. It’s called Weekend Testers. It seems to be the beginning of a truly Indian version of the Context-Driven School of Testing. It’s a sapient testing insurgency!

If you are a tester in India (or you aspire to be) and you want to be a GOOD tester, then you should get involved with this or start something like it yourself.

I taught in India, years ago, planting seeds at Wipro and Mindtree. Michael Bolton has taught there a few times, since. I’ve been working with Pradeep Soundararajan and Shrini Kulkarni for a while. I just discovered that Matthew Heusser has also been mentoring some of them, too. I’ve blogged about other Indians who test. But this is the first organized community of skill-bearing software testers I have seen in India.

I hope it’s the beginning of something big. You deserve it. Plus… when there’s enough of you, and some of you get into management, perhaps you’ll invite me to come out and teach again. I’d like to go back for more.

Michael Bolton, my brother Jon, and I have produced a new version of our Exploratory Testing Dynamics document. We unveiled it last week at STPCON.

This document describes the elements of exploratory testing as we currently understand them. This new version has not yet been reviewed by Cem Kaner or any of our colleagues who normally have opinions on this, but I don’t anticipate that it will be revised much when they do. It should be fairly stable, for now.

ET cannot be summed up fairly as “unstructured playing around” and if you look at this document you will see why. If nothing else, count the elements in it. Crikey.

The document consists of four lists:

1. Evolving Work Products

Exploratory testing is an approach that emphasizes evolution. We start with something and make it better and better. Often testers can be overly focused on finding bugs, but look at all the things that get better. For instance, you can run your automated regression tests a thousand times and the tests don’t learn and neither do the button pushing testers. But you learn a lot by sapiently retesting– you evolve. One of the products of ET is a better tester.

Some evolution also happens in highly scripted testing, but in ET it’s a primary goal. It’s the point.

2. Skills and Tactics

What does an exploratory tester do? Well, feast your eyes on the Skills and Tactics list. Each of those items are things that seem to us to be reasonably independent, teachable, and observable. Some of them are also skills that a highly scripted tester would need, but they work a little differently for explorers. For instance, the skills of reading and analyzing documents means doing that to prepare for or aid exploration of the product, rather than to prepare detailed instructions for scripted test execution.

3. ET Polarities

The Polarities list is unique to ET. An exploratory tester must manage his own mental process, and that process is about developing ideas and using those ideas to search the product for trouble. We’ve found over years of experience doing this that a tester loses steam if he spends too long in one train or type of thought. So, what can he do?

One answer is alternation. We alternate among complementary activities. Two activities are complementary if performing one of them rejuvenates the other. A great example is reading about a product and directly interacting with it.

4. Test Strategy

The Heuristic Test Strategy model is a set of guideword heuristics that we publish as part of the Rapid Software Testing class. In addition to the general ideas of exploration, evolution, and alternation, the guidewords represent the specific subject matter of testing– the conditions we need to succeed, the things we look at, the things we look for, and the specific test techniques we use to produce tests.

With these lists, I can systematically evaluate a tester and identify strengths and weaknesses. We in the Context-Driven School of testing need models like this, because we focus on skills, rather than techniques and tools. This document allows us to focus our efforts and develop specific exercises for tester training.

An open letter to James Whittaker:

You wrote: “I had an amicable hallway conversation with James Bach. His blogger angst at my use of the title ‘Exploratory Testing’ didn’t spill over to a face-to-face discussion. Frankly, I am not surprised. I’ve never claimed the term as my own, I simply took it and made it work in the hands of real testers on real software under real ship pressure. Consultants can coin all the terms they want, but when us practitioners add meat to their pie, why cry foul? Is it not a better reaction to feel happy that there are people actually doing something with the idea?”

None of that is true.

I would not describe our conversation as amicable. Perhaps you thought it was amicable because we didn’t talk about anything important, and during that moment, I didn’t raise my voice. Or punch you.

My criticism of you is not “blogger angst”, it’s my opinion based on studying for 20 years something you’ve hardly studied at all. Every substantive conversation we’ve had has consisted of you denying whatever I happen to say, without offering evidence and in most cases without offering an argument. You have a zeal for dismissing my work that is truly extraordinary– you once even denied, again without evidence, that I knew how to run a file compare tool. Wow.

Now you say you made ET work? Well, first, you don’t know what ET is. Second, you’re an academic. You stayed in school and studied formal methods (that no one uses) while I was cutting my teeth in Silicon Valley. I have taught and demonstrated ET all over the world. I’m not alone, but work with a community of like-minded testers and thinkers, comparing notes with them, and deepening our understanding of exploratory learning applied to testing. You have not been a part of that.

I don’t think you’re adding meat, I think you’re serving thin gruel.

ET does work. My community repeatedly shows that it does. We will patiently continue to teach and develop it.

Subtlety hasn’t worked with you. So I’m saying this publicly: You’re a good speaker, but as a practitioner, if your prose is any indication, you don’t know much about what you are doing. If you applied yourself, you could become a good tester. But I’ve seen no evidence of that, yet.

When Cem Kaner introduced the term “exploratory testing” in the mid-80’s, everyone ignored it. When I picked up the term and ran with it, I was mostly ignored. But slowly, it spread through the little community that would become the Context-Driven School. I began talking about it in 1990, and created the first ET class in 1996. It wasn’t until 1999 that Cem and I looked around and noticed that people who were not part of our school had begun to speak and write about it, too.

When we looked at what some of those people were saying, yikes! There was a lot of misunderstanding out there. So, we just kept plugging along and running our peer conferences and hoping that the good would outweigh the bad. I still think that will happen in the long run.

But sometimes it’s hard to stomach how the idea gets twisted. Case in point: James Whittaker, an academic who has not been part of the ET leadership group, and also has little or no experience on an industrial software project as a tester or test manager, has published a book called Exploratory Software Testing.

Whatever Whittaker means when he talks about exploratory testing is NOT what those of us mean who’ve been working on nurturing and developing ET for the last 20 years. As far as I can tell, he has not made more than a shallow study of it. I will probably not write a detailed review (though his publisher asked me to look at it before it was published), because I get too angry when I talk about it, and I would rather not be angry. But Adam Goucher has published his review here.

Another guy who shows up at the conferences, B.J. Rollison, also gets ET wrong. He’s done what he calls “empirical research” into ET, at Microsoft. Since he, again, has not engaged the community that first developed the concept and practices of ET, it’s not altogether surprising that his “research” is based on a poor understanding of ET (for instance, he insists that it’s a technique, not an approach. This is similar to confusing the institution of democracy with the mechanics of voting), and apparently were carried out with untrained subjects, since Rollison himself is not trained in what I recognize as exploratory testing skills.

Experimental research into ET can be done, but of course any such work is in the realm of social science, not computer science, because ET is a social and psychological phenomenon. (see the book Exploring Science, for an example of what such research looks like).

Now even within the group of us who’ve been sharing notes, debating, and discovering the roots of professional exploratory thinking in the fields of epistemology and cognitive psychology and the philosophy and study of scientific practice, there are strong differences of opinion. There are people I disagree with (or who just dislike me) whom I still recognize as thoughtful leaders in the realm of exploratory testing (James Lyndsay and Elisabeth Hendrickson are two examples). Perhaps Whittaker and Rollison will become rivals who make interesting discoveries and contributions, at some point. Time will tell. Right now, in my opinion, they are just skating on the surface of this subject.

I told James Whittaker that I don’t use the term “manual testing” and that I prefer the term “sapient testing” because it’s more to the point. This is evident in the first definition of the word “manual” in the Oxford English Dictionary: ” 1. a. Of work, an action, a skill, etc.: of or relating to the hand or hands; done or performed with the hands; involving physical rather than mental exertion. Freq. in manual labour.“ Sapient, on the other hand, means “wise.”

Whittaker laughed and said “Bach, you are always making words up.” And then told me that in his opinion manual testing did not evoke the concept of unskilled manual labor. Now, other than establishing that James Whittaker doesn’t have an online account to the O.E.D. (definition of “sweeeeet!” is “an online O.E.D. account.”), or perhaps doesn’t consider dictionaries to be useful sources of information about what words mean, I see something else in his reaction: I blew his mind. What I said doesn’t intersect with anything in his education.

To understand me, Whittaker will have to use his sapience, rather than responding manually (i.e. with his hands).

In other words, I notice that some of my rivals in the testing industry don’t merely disagree with me, they apparently don’t comprehend what I’m saying. Example: after some ten hours of solid debate with me, over several sittings, Stuart Reid (who is working on a software testing standard of all preposterous things), told a colleague of mine that he believed I don’t truly mean the things I said in those debates, but merely said them to “be provocative.” Huh. That’s some serious cognitive dissonance you got going, Stu, when the only way you can process what I’m saying is to declare, essentially, that it was all just a dream.

Of course, I don’t think this is an intelligence problem. I think this is a lack-of-effort-to-use-intelligence problem. It’s not convenient for certain consultants to tear up their out-of-date books and classes and respond to the challenge of, um, the last 30 years of development of the craft. So they continue to teach and preach ideas from the seventies (or create testing standards based on them, because they believe not enough people appreciate testing disco).

Anyway, in the Context-Driven community’s latest attempt to explain the ins and outs and vital subtleties of testing, Michael Bolton has come up with a promising tack. Maybe this will help. He’s drawing a distinction between testing and checking.

Brace yourselves for insight. A lot of what people call testing is actually mere checking. But even checking requires testing intelligence to design and do well. This gives more specifics to my concept of sapient testing. Here are Michael’s seminal posts on the subject:

1. http://www.developsense.com/2009/08/testing-vs-checking.html
2. http://www.developsense.com/2009/09/transpection-and-three-elements-of.html
3. http://www.developsense.com/2009/09/pass-vs-fail-vs-is-there-problem-here.html
4. http://www.developsense.com/2009/09/elements-of-testing-and-checking.html

When Michael first made the distinction between testing and checking, I was annoyed. Truly. It blew my mind in that bad way. I thought he was manufacturing a distinction that we didn’t need. I decided to ignore it. Then he called me and asked “So what do you think of my checking vs. testing article?” I had to say I didn’t like it at all. We argued…

…and he convinced me that it was a good idea. Thank you dialectical learning! Debate has saved me again!

I now agree that it’s a practical distinction that can be used as a lens to focus on the quality of a test process. I do have to get used to the words, though. I now see a difference between automated testing and automated checking, for instance: automated testing means testing supported by tools; automated checking means specific operations, observations, and verifications that are carried out entirely with tools. Automated testing may INCLUDE automated checking as one component, however automated checking does NOT include testing.

Making this distinction is exactly like distinguishing between a programmer and a compiler. We do not speak of a compiler “writing a program” in assembly language when it compiles C++ code. We do not think that we can fire the programmers because the compiler provides “automated programming.” The same thing goes for testing. Or… does that blow your mind?

Context-driven testing is apparently very difficult to understand. I wouldn’t have thought that it’s a difficult concept, except for the last decade I’ve watched hundreds of people in terrible confusion. Testers outside of the United States have an especially hard time with it, for some reason that escapes me.

I do have some working theory of the misunderstanding. I think it has a lot to do with how so many people are looking for first-order formulaic answers to every problem. Context-driven thinking rejects that. We say you can’t do quality work unless you develop skill. There’s no secret phrase or recipe book that will grant you skill. You must develop skill by struggling with testing problems.

India has a lot of testers, but so very few have shown publicly that they can think outside the “best practices” and Factory School kiddie pool. But, I’m happy to say, Parimala Shankaraiah is one of them. I’m adding her to my blogroll. Her blog is called Curious Tester. (It’s nice to be able to honor a female tester for a change.)

Bear in mind I have no idea if she thinks of herself as context-driven. She hasn’t mentioned it directly. I believe she is, based on how she writes.

Several things strike me about Pari. One is her sheer enthusiasm. Her love of testing is infectious. Another is the breadth of her imagination. In one post she related the book The Secret to her experience as a tester. That was impressive analogizing! Almost acrobatic. She believes in peer conferences and she opposes certification, too. Those are important indicators of a context-driven sensibility.

I have often said that I expect the Indian testing industry to wake up someday (at least a few thousand of them) and realize that they can do a lot better than writing dimwitted scripted test cases. Pari is hopefully just the beginning of the Indian testing renaissance.

When my publisher decided to release my Secrets book for “free” as an ebook, I thought it meant FREE. I just found it meant free-for-a-little-while-and-then-gone. Apparently it has some sort of DRM expiration date on it.

[Update: One of my tester friends found that he could defeat the expiration mechanism by playing with the system date, but that's not very practical, I guess.]

They didn’t mention this to me when they told me about the promotion. I suppose they thought “Well OBVIOUSLY it’s only free for a period of time… we’re in the publishing business!” But it wasn’t obvious to me or I would have told you when I announced it.

Anyway, you can still download the book until the 24th of July, and you can read it for some period of time after that before it goes POOF. Not bad, considering that it didn’t cost you anything except the annoyance of downloading it, but still…

Naturally, I hope everyone goes out and buys the hardcover with real money. That way I’ll be encouraged to write more books. Otherwise, it’s an expensive hobby.

Side note: It’s interesting how glitchy the process is, too. Several people found that the special Adobe reader software complained and moaned and wouldn’t install on their systems. QUALITY IS DEAD!