This post is co-authored with Michael Bolton. We have spent hours arguing about nearly every sentence. We also thank Iain McCowatt for his rapid review and comments.

Testing and tool use are two things that have characterized humanity from its beginnings. (Not the only two things, of course, but certainly two of the several characterizing things.) But while testing is cerebral and largely intangible, tool use is out in the open. Tools encroach into every process they touch and tools change those processes. Hence, for at least a hundred or a thousand centuries the more philosophical among our kind have wondered “Did I do that or did the tool do that? Am I a warrior or a just spear throwing platform? Am I a farmer or a plow pusher?” As Marshall McLuhan said “We shape our tools, and thereafter our tools shape us.”

This evolution can be an insidious process that challenges how we label ourselves and things around us. We may witness how industrialization changes cabinet craftsmen into cabinet factories, and that may tempt us to speak of the changing role of the cabinet maker, but the cabinet factory worker is certainly not a mutated cabinet craftsman. The cabinet craftsmen are still out there– fewer of them, true– nowhere near a factory, turning out expensive and well-made cabinets. The skilled cabineteer (I’m almost motivated enough to Google whether there is a special word for cabinet expert) is still in demand, to solve problems IKEA can’t solve. This situation exists in the fields of science and medicine, too. It exists everywhere: what are the implications to skilled human work of the evolution of tools? Anyone who seeks excellence in his craft must struggle with the appropriate role of tools.

Therefore, let’s not be surprised that testing, today, is a process that involves tools in many ways, and that this challenges the idea of a tester.

This has always been a problem– I’ve been working with and arguing over this since 1987, and the literature of it goes back at least to 1961– but something new has happened: large-scale mobile and distributed computing. Yes, this is new. I see this is the greatest challenge to testing as we know it since the advent of micro-computers. Why exactly is it a challenge? Because in addition to the complexity of products and platforms which has been growing steadily for decades, there now exists a vast marketplace for software products that are expected to be distributed and updated instantly.

We want to test a product very quickly. How do we do that? It’s tempting to say “Let’s make tools do it!” This puts enormous pressure on skilled software testers and those who craft tools for testers to use. Meanwhile, people who aren’t skilled software testers have visions of the industrialization of testing similar to those early cabinet factories. Yes, there have always been these pressures, to some degree. Now the drumbeat for “continuous deployment” has opened another front in that war.

We believe that skilled cognitive work is not factory work. That’s why it’s more important than ever to understand what testing is and how tools can support it.

Checking vs. Testing

For this reason, in the Rapid Software Testing methodology, we distinguish between aspects of the testing process that machines can do versus those that only skilled humans can do. We have done this linguistically by adapting the ordinary English word “checking” to refer to what tools can do. This is exactly parallel with the long established convention of distinguishing between “programming” and “compiling.” Programming is what human programmers do. Compiling is what a particular tool does for the programmer, even though what a compiler does might appear to be, technically, exactly what programmers do. Come to think of it, no one speaks of automated programming or manual programming. There is programming, and there is lots of other stuff done by tools. Once a tool is created to do that stuff, it is never called programming again.

Now that Michael and I have had over three years experience working with this distinction, we have sharpened our language even further, with updated definitions and a new distinction between human checking and machine checking.

First let’s look at testing and checking. Here are our proposed new definitions, which soon will replace the ones we’ve used for years (subject to review and comment by colleagues):

Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.

(A test is an instance of testing.)

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

(A check is an instance of checking.)

Explanatory notes:

• “evaluating” means making a value judgment; is it good? is it bad? pass? fail? how good? how bad? Anything like that.
• “evaluations” as a noun refers to the product of the evaluation, which in the context of checking is going to be an artifact of some kind; a string of bits.
• “learning” is the process of developing one’s mind. Only humans can learn in the fullest sense of the term as we are using it here, because we are referring to tacit as well as explicit knowledge.
• “experimentation” implies interaction with a subject and observation of it as it is operating, but we are also referring to “thought experiments” that involve purely hypothetical interaction. By referring to experimentation, we are not denying or rejecting other kinds of learning; we are merely trying to express that experimentation is a practice that characterizes testing. It also implies that testing is congruent with science.
• the list of words in the testing definition are not exhaustive of everything that might be involved in testing, but represent the mental processes we think are most vital and characteristic.
• “algorithmic” means that it can be expressed explicitly in a way that a tool could perform.
• “observations” is intended to encompass the entire process of observing, and not just the outcome.
• “specific observations” means that the observation process results in a string of bits (otherwise, the algorithmic decision rules could not operate on them).

There are certain implications of these definitions:

• Testing encompasses checking, whereas checking cannot encompass testing.
• Checking is a process that can, in principle be performed by a tool instead of a human, whereas testing can only be supported by tools. Nevertheless, tools can be used for much more than checking.
• Testing is an open-ended investigation– think “Sherlock Holmes”– whereas checking is short for “fact checking” and focuses on specific facts and rules related to those facts.
• Checking is not the same as confirming. Checks are often used in a confirmatory way (most typically during regression testing), but we can also imagine them used for disconfirmation or for speculative exploration (i.e. a set of automatically generated checks that randomly stomp through a vast space, looking for anything different).
• One common problem in our industry is that checking is confused with testing. Our purpose here is to reduce that confusion.
• A check is describable; a test might not be (that’s because, unlike a check, a test involves tacit knowledge).
• An assertion, in the Computer Science sense, is a kind of check. But not all checks are assertions, and even in the case of assertions, there may be code before the assertion which is part of the check, but not part of the assertion.
• These definitions are not moral judgments. We’re not saying that checking is an inherently bad thing to do. On the contrary, checking may be very important to do. We are asserting that for checking to be considered good, it must happen in the context of a competent testing process. Checking is a tactic of testing.

Whither Sapience?

If you follow our work, you know that we have made a big deal about sapience. A sapient process is one that requires an appropriately skilled human to perform. However, in several years of practicing with that label, we have found that it is nearly impossible to avoid giving the impression that a non-sapient process (i.e. one that does not require a human but could involve a very talented and skilled human nonetheless) is a stupid process for stupid people. That’s because the word sapience sounds like intelligence. Some of our colleagues have taken strong exception to our discussion of non-sapient processes based on that misunderstanding. We therefore feel it’s time to offer this particular term of art its gold watch and wish it well in its retirement.

Human Checking vs. Machine Checking

Although sapience is problematic as a label, we still need to distinguish between what humans can do and what tools can do. Hence, in addition to the basic distinction between checking and testing, we also distinguish between human checking and machine checking. This may seem a bit confusing at first, because checking is, by definition, something that can be done by machines. You could be forgiven for thinking that human checking is just the same as machine checking. But it isn’t. It can’t be.

In human checking, humans are attempting to follow an explicit algorithmic process. In the case of tools, however, the tools aren’t just following that process, they embody it. Humans cannot embody such an algorithm. Here’s a thought experiment to prove it: tell any human to follow a set of instructions. Get him to agree. Now watch what happens if you make it impossible for him ever to complete the instructions. He will not just sit there until he dies of thirst or exposure. He will stop himself and change or exit the process. And that’s when you know for sure that this human– all along– was embodying more than just the process he agreed to follow and tried to follow. There’s no getting around this if we are talking about people with ordinary, or even minimal cognitive capability. Whatever procedure humans appear to be following, they are always doing something else, too. Humans are constantly interpreting and adjusting their actions in ways that tools cannot. This is inevitable.

Humans can perform motivated actions; tools can only exhibit programmed behaviour (see Harry Collins and Martin Kusch’s brilliant book The Shape of Actions, for a full explanation of why this is so). The bottom line is: you can define a check easily enough, but a human will perform at least a little more during that check– and also less in some ways– than a tool programmed to execute the same algorithm.

Please understand, a robust role for tools in testing must be embraced. As we work toward a future of skilled, powerful, and efficient testing, this requires a careful attention to both the human side and the mechanical side of the testing equation. Tools can help us in many ways far beyond the automation of checks. But in this, they necessarily play a supporting role to skilled humans; and the unskilled use of tools may have terrible consequences.

You might also wonder why we don’t just call human checking “testing.” Well, we do. Bear in mind that all this is happening within the sphere of testing. Human checking is part of testing. However, we think when a human is explicitly trying to restrict his thinking to the confines of a check– even though he will fail to do that completely– it’s now a specific and restricted tactic of testing and not the whole activity of testing. It deserves a label of its own within testing.

With all of this in mind, and with the goal of clearing confusion, sharpening our perception, and promoting collaboration, recall our definition of checking:

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

From that, we have identified three kinds of checking:

Human checking is an attempted checking process wherein humans collect the observations and apply the rules without the mediation of tools.

Machine checking is a checking process wherein tools collect the observations and apply the rules without the mediation of humans.

Human/machine checking is an attempted checking process wherein both humans and tools interact to collect the observations and apply the rules.

In order to explain this thoroughly, we will need to talk about specific examples. Look for those in an upcoming post.

Meanwhile, we invite you to comment on this.

UPDATE APRIL 10th: As a result of intense discussions at the SWET5 peer conference, I have updated the diagram of checking and testing. Notice that testing is now sitting outside the box, since it is describing the whole thing, a description of testing is inside of it. Human checking is characterized by a cloud, because its boundary with non-checking aspects of testing is not always clearly discernible. Machine checking is characterized by a precise dashed line, because although its boundary is clear, it is an optional activity. Technically, human checking is also optional, but it would be a strange test process indeed that didn’t include at least some human checking. I thank the attendees of SWET5 for helping me with this: Rikard Edgren, Martin Jansson, Henrik Andersson, Michael Albrecht, Simon Morley, and Micke Ulander.

My sister, Erica, is not a programmer. Normally she’s not a tester, either. But recently she paired with me, playing a tester role, and spotted bugs while I wrote in Perl. In the process, it became clear to me that testers do not need to become programmers in order to help programmers write programs in real-time.

The Context

While working on the report for the Rapid Testing Intensive, recently, I needed a usable archive of the materials. That meant taking all of the pages, comments, and attachments out of my Confluence site and putting them in a form easier to shuffle, subdivide, organize, refer to, and re-distribute. It would be great if that were a feature of Confluence, but the closest I can get to that is either manually downloading each item or downloading an entire archive and dealing with a big abstract blob of XML and cryptically named files with no extensions.

(Note to Atlassian: Please enhance Confluence to include a archivist-friendly (as opposed to system administrator-friendly) archive function that separates pages, attachments, and comments into discrete viewable units with reasonable names.)

The Deflection

While Erica catalogued the names of all the attachments representing student work and the person or persons who created them, I was supposed to write a program to extract the corresponding material from the archive. Instead, I procrastinated. I think I checked email, but I admit it’s possible I was playing Ghost Recon or watching episode 13 of Arang and the Magistrate on Hulu. So, when she was ready with the spreadsheet, I hadn’t even started my program.

To cover my laziness, I thought I’d invite her to sit with me while I wrote it… you know, as if I had been waiting for her on purpose to show her the power of code or whatever. I expected her to decline, since like many computer power users, she has no interest in programming, and no knowledge of it.

The Surprising Outcome

She did not decline. She sat with me and we wrote the program together. She found six or seven important bugs while I typed, and many other little ones. The programming was more fun and social for me. I was more energized and focused. We followed up by writing a second, bigger program together. She told me she wants to do more of this kind of work. We both want to do more.

A Question

How does someone who knows nothing about Perl programming, and isn’t even a tester, productively find bugs almost immediately by looking at Perl code?

That’s kind of a misleading question, because that’s not what really happened. She didn’t just look at my code. She looked at my code in the context of me talking to her about what I was trying to do as I was trying to do it. The process unfolded bit by bit, and she followed the logic as it evolved. It doesn’t take any specific personal quality on the part of the “coding companion,” just general ones like alertness, curiosity, and basic symbolic intelligence. It doesn’t take any particular knowledge, although it can help a lot.

Perhaps this would not work well for all kinds of coding. We weren’t working on something that required heaps of fiddly design, or hours of doodling in a notebook to conceive of some obscure algorithm.

My Claim

A completely non-technical but eager and curious companion can help me write code in real-time by virtue of three things:

1. The dynamic and interactive legibility of the coding process. I narrate what I’m doing as it comes together step-by-step. The companion doesn’t eat the whole elephant in a bite; and the companion encounters the software mediated by my continuous process of interpretation. I tell him what and why and how. I do this repeatedly, and answer his questions along the way. This makes the process accessible (or in the parlance I like to use “legible” because that word specifically means the accessibility of information). The legibility is not that of a static piece of code, sitting there, but rather a property of something that is changing within a social environment. It’s the same experience as watching a teacher fill a whiteboard with equations. If you came at the end of the class, it would look bewildering, but if you watched it in process, it looks sensible.
2. The conceptual simplicity of many bugs. Some bugs are truly devious and subtle, but many have a simple essence or an easily recognized form. As I fix my own bugs and narrate that process, my coding companion begins to pick up on regularities and consistency relationships that must be preserved. The companion programs himself to find bugs, as I go.
3. The sensemaking faculties of a programmer seeking to resolve the confusion of a companion. When my dogs bark, I want to know why they are barking. I don’t know if there’s a good reason or a bad reason, but I want to resolve the mystery. In the course of doing that, I may learn something important (like “the UPS guy is here”). Similarly, when my coding companion says “I don’t understand why you put the dollar sign there and there, but not over there” my mind is directed to that situation and I need to make sense of it. It may be a bug or not a bug, but that process helps me be clear about what I’m doing, no matter what.

And Therefore…

A tester of any kind can contribute early in a development process, and become better able to test, by pairing with a programmer regardless of his own ability to code.

There’s a reason I haven’t been blogging much. Every time I want to blog about something I remember that I should be working on finishing my report from the first Rapid Testing Intensive. Lots of reasons that’s been delayed (lots of work on the road plus my father being almost dead for weeks temporarily destroyed my concentration), but now I have something to say about it. There’s been progress.

The Rapid Testing Intensive Onsite happened in July on Orcas Island. This training event involved more than 100 testers, 20 of which were here on Orcas Island, and 80+ scattered around the world. We decided afterward that it was overambitious to do an onsite/online simultaneous seminar, but we did collect a lot of student work and other data. One of the things I promised was to make this public. A grand test report.

Turns out that’s a lot of work. I wrote software last week to analyze chatroom traffic (to help study how chatrooms might have helped remote testers work together). I also wrote code to extract comments and hundreds of attachments from our Confluence archive in order to gather the raw data for reporting. We are chipping away at the problem. Actually we did another RTI online just recently– a smaller scale event– and the report from that has already been delivered to the students.

In the process I found a bug in Confluence. Atlassian nicely allows you to backup your entire Confluence archive. But then its servers won’t let you actually download it, because they automatically terminate long downloads. Then when it terminated, I couldn’t try again because the download link disappeared and the system informed me that I could only have one download per day. I don’t know how any reasonable sized customer has ever been able to download their archive without using special tools. Their customer service people have offered to slice up the zip files, next time, but what I don’t understand is why they are offering a workaround to this instead of just fixing the problem. It’s a pretty important feature to get your backup.

Despite my troubles with Confluence, we are pretty happy with it– that’s down to great customer service.

Chatroom Analysis

Part of why I want to do testing intensives is to collect empirical data on exploratory testing. For instance, I’m curious about how a chatroom might help a group of remote testers collaborate. So we used HipChat for communication– an Atlassian product I am quite happy with– then mined and fixed up the message archive (more than 10,000 messages I extracted and reformatted and in some cases tweaked by hand). I then used NodeXL from Microsoft (yet another unreliable tool with wonderful capabilities from the boys in Redmond) to perform a network analysis of the directed chat messages.

The lines represent directed messages (typing in a chat room but using a person’s handle to get their attention). The fuchsia lines represent communication between students. Gray lines are communications to or from the staff. Blue nodes are teaching staff; green is admin; and black are students. From this I can see that there was a huge variation among the students in how they used the chatrooms. The offsite students used it a lot more, in general, as I would expect. A few students were heavy talkers, connecting with a lot of other students. Based on this analysis, I’ve invited Griffin Jones to be a peer advisor at the next RTI, since he was one of the most active talkers, helping to create a good social atmosphere.

I would really like to do a semantic analysis by tagging each message with a category. Were people being generally social or were they actually talking about testing? That analysis may have to wait, but at least we have the data.

Someone asked for a word cloud. I don’t find those very interesting, but here it is anyway.

I’m wrapping up 91 days on the road. I’ve taught around Europe, become a mostly-vegetarian, lost 23 pounds, and now I’m doing my last day in India.

India. OMG. I have never had such a thunderous reception. I was here nine years ago, taught two classes, and learned that Indians can be fantastic testers– if only they would wake themselves up to do it. Since then, hundreds of them have begun to do that.

Last time I visited Mindtree, I taught my class to a skeptical audience with good potential, then I went home. This time I met with the CEO and COO, spoke to a forum of 250 testers, did a three-camera interview, dined with their managers while monkeys (real, literal monkeys, not bad testers) screeched and cavorted nearby.

I visited Moolya, the hottest new test lab anywhere, and brought their work to a halt while they presented their projects for my review. I spent most of a week coaching their head of training, Parimala, about whom I have blogged before.

I taught at Intel, which was more receptive than I expected they would be.

I’m wrapping up at Barclays, in Pune, which is veritably vibrating with creative intelligence, ready to break free. Keith Klain and his head of training Leah Stockley are responsible for making that happen.

India may be the new garden of the skilled testing revolution. Behold the harvest.

I’ll write a more detailed report for Tea Time with Testers. Stay tuned.

We recently ran the first Rapid Testing Intensive on Orcas Island. One of the outcomes of this, as I have promised, is a public test report on what happened. This is my report report. I am reporting on my reporting…

What was the “Rapid Testing Intensive”?

Oh, it was just the most realistic and ambitious test training event ever. Sorry if you missed it. We (17 testers onsite and nearly ninety online) tested a part of eBay for four solid days, finding many important bugs. But it was more about training and coaching than bug finding.

Where am I with this report?

I am actually creating several reports. One is a report about our process. This is meant for people curious about Rapid Testing methodology, and in it I will contrast what we did with the “traditional” approach. I will also create a report for our primary client, eBay, whose software we tested.

There’s a lot to do and I am plugging away. I just finished the usability testing roll-up report. I’m going through 320+ bug reports now. Here’s my overall todo list:

Analyze HipChat traffic

• who was talking most?
• who was being responded to most?
• when were people talking most?
• were conversations more social or technical?
• what proportion of the conversation were people seeking information, and what proportion were people providing information?
• was HipChat helpful in coordinating a worldwide testing effort?

Analyze bug list

• Assess bug report quality
• Reproduce bugs
• Close duplicates and non-bugs
• Add labels and categorize according to risk
• What did eBay know about before we started?
• What does eBay consider important that we found?

Produce “official” test coverage outline

• Analyze student TCOs

Produce “official” Risk List

Produce official sequence of events

Describe our test activities

• Analyze student test session reports

Describe what we did not test

Analyze tools used

Identify outstanding student work

Write report narrative and summary

How “Rapid” is this?

Yeah, this seems like a lot of work and yes it will take a while (I’m working evenings on it while teaching during the day). But this is because it’s a seminar and demonstration project. In a real project, our reporting (other than the bug list itself) is either oral, or expressed in brief written statements. On a normal project, once we get organized (after a few days if it’s a new project) reporting gets pretty easy.

I’m excited to be working with Barclays on an unprecedented project: creating a professional testing culture based on the Context-Driven principles and my Rapid Software Testing (RST) methodology. The Barclays Global Test Centre (GTC), led by Keith Klain, has hundreds of testers spread around the world. They work in a regulated industry on high stakes products. But unlike nearly every other large organization in the world, they have decided not to rely on pretense and 40 year-old ideas that were discredited 30 years ago. They are instead putting in place a system to recruit and grow highly skilled and highly motivated testers.

Barclays’ approach in the GTC is to identify and encourage dozens of testing champions in its ranks who are the role models and mentors for the rest of the group. Anyone may aspire to be in this special group, but to be recognized requires that the candidate tester demonstrate vigorous self-education and critical analysis. Some of the testers in the group began as strong skeptics of Rapid Testing. But the methodology is designed for skeptics– it is based on skill development and heuristics rather than pushing “best practices.” In Rapid Testing, the skilled tester is always in charge, not pieces of paper or officious charts.

RST requires each tester to employ his own judgment and technical analysis, much like what airlines expect of pilots, or hospitals expect of doctors. That can’t work on a large scale without a strong corporate commitment to training and personal ethics. Management must drive out fear, so that testers are willing to take the sort of risks that come from making their own decisions about test strategy. But the onus is on the testers to earn personal credibility within an internal community that can effectively police itself. Any tester, at any time, is expected to stand up and explain and defend his work.

I’m aware of only two large companies in the world that have made a commitment to this kind of professionalism, which is an altogether different sort of professionalism than the ceremonial certification variety that is promoted by most organizations. In Barclays’ case, this commitment has strong support from top management, and I have personally witnessed, in my weeks of working with them, that the testers at their Singapore operation have fire in their eyes. There are testers here who deserve to have an international reputation.

I am thinking and talking a lot about ethics, lately. Maybe that’s because Context-Driven testing is getting better traction, now. More testers are approaching me, asking for guidance. More testers are challenging fake testing in their organizations.

Or maybe I’m just noticing it more, because Cem Kaner used to take the brunt of all this. In that respect, I have historically been more a follower than a leader in our community. Recently, I’ve stepped forward to be more of the role model that I am expected by my colleagues to be.

(Note: Ethics is a guaranteed sore subject. Whenever I talk about ethics, I can’t avoid implying that I believe some people are not as ethical as they should be. But, you know, that’s the burden of leadership. To those who avoid politically difficult subjects, enjoy your shadowy existence. Lurk on, yon lurkers.)

I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.

I need my own, because these codes above don’t directly deal with certain common testing ethics issues.

So, I’m the content-owner for the 2nd Kiwi Workshop on Software Testing, tomorrow morning, here in Wellington, New Zealand. Our topic is ethics. To get ready for it, I thought I would write out some of the ethical principles by which I strive to operate. Here they are:

• Know what a test is. Avoid labeling an activity as a “test” unless it represents a sincere effort to discover a problem in a product.
• Maintain a reasonable impartiality. The purpose of testing is to cast light on the status of the product and its context, in the service of my clients. I may play multiple roles on a project, but my purpose, insofar as I am a tester,  is not to design or improve the product.
• Do not claim to assure, ensure, or control quality. I don’t control anything about the product: a tester is a witness. In that capacity, I strive to assist the quality creation process.
• Report everything that I believe, in good faith, to be a threat to the product or to the user thereof, according to my understanding of the best interests of my client and the public good.
• Apply test methods that are appropriate to the level of risk in the product and the context of the project.
• Alert my clients to anything that may impair my ability to test.
• Recuse myself from any project if I feel unable to give reasonable and workman-like effort.
• Make my clients aware, with alacrity, of any mistake I have made which may require expensive or disruptive correction.
• Do not accede to requests by my client to work in a wasteful, dangerous, or deceptive way. (e.g. I will not keep test case metrics, because they are damaging in almost any context)
• If I do not understand or accept my mission, it shall be my urgent priority to discover it or renegotiate it.
• Do not deceive my clients about my work, nor help others to perpetrate deception.
• Do not accept tasks for which I am not reasonably prepared or possess sufficient competence to perform, unless I am under the direction and supervision of someone who can guide me.
• Study my craft. Be alert to better solutions and better ways of working.

For the first time in a long time, I am prepared to endorse a new trainer of the Rapid Software Testing class: Paul Holland.

Paul has been a tester and test manager at Alcatel/Lucent for something like 17 years. In recent years he has been using and teaching Rapid Software Testing methodology within Alcatel. His practical experience is fabulous.

But that’s not why I’m happy to have him working for me. There are a lot of experienced people in the world.

He is a pilot. He flew Sea King helicopters for the Canadian Navy. The thing about pilot training is that it is similar to good tester training. You have to practice, practice, practice. I’m a pilot, too. Most of my family flies airplanes. My approach to training is much influenced by that experience.

Still, that’s not really why I’m excited.

I’m excited because Paul enjoys an excellent reputation within the Context-Driven community as a facilitator of peer conferences. He’s irreverent. He’s an innovator. That’s important, because anyone who teaches my class must make it his own class, too. Paul has the right temperament for that.

Paul’s teaching style is a bit less aggressive than mine (as is Michael Bolton’s), if you like that sort of thing. He will teach through Satisfice, Inc., so if you want him to do a class at your site, contact me.

Bharani from India is a novice tester who appears to have a lot of energy. She recently emailed me this:

Sir, I need one small tips. As a, new tester how should they improve to became sharp and clever in the testing field.

 I mean that, What is their homework. what they should do daily…without skipping. 

For ex: like normal daily activities. we have to do exercise  breakfast..so….so..

I mean that, what is the testers daily activities?

I am asked many questions by testers. This particular one reveals a lot about the questioner. For one thing, she thinks differently, because I have never been asked this. And it’s a question about discipline, so perhaps she’s a tester who is truly serious about developing herself.

My first answer is the same as the one my world famous writer of a father so often gives: write.

Write every day.

For me this takes the form of carrying my Moleskine notebook everywhere I go (now with a bandolier for my pens!). Whenever I find myself with a few moments, I make notes of my thoughts about testing and technical life. My notebooks serve as one source of great new ideas for my consulting and teaching.

But for testers, there are a few more practices to keep in mind…

Watch yourself think ever day.

While you are working, notice how you think. Notice where your ideas come from. Try to trace your thoughts. This is not easy. You have to practice to improve your skill of self-observation.

As a tester, you must become an artist of your thoughts, you must learn to see the structure and gain control of the structures by which you notice what no one else notices. These structures are not simply “intuition” or “common sense.” We can do better than that.

Question something about how you work every day.

Testers question things, of course. That’s what testing is. But too few testers questions how they work. Too few testers question why testing is the way it is.

For example: Why do we talk about tests “passing” and “failing”, instead of talking about whether there are problems? Do all bugs need to have steps to reproduce? How do we decide when to write a test down and what steps need to be written?

Explain testing every day.

The ability to explain testing is linked to nearly every other testing skill. Even if no one makes you explain your methodology, you can explain it to yourself. Do that. Practice doing it by voice and in writing. Don’t rest with buzzwords. Don’t tell yourself that you did “black box testing.” Break it down. Explain what you did specifically and explain why you did it.

I’m delighted that Rob Sabourin, creator of the Just-In-Time Testing class, will be with us during the Rapid Testing Intensive, in July. He’s coming as a student, but you know, we’re ALL students of the craft, so I’m going to put him to work. Big time. We will also have Ajay Balamurugadas onsite. He’s among the most famous of the Indian Context-Driven testers.

As I figured, most of the sign-ups have been for online participation. So far, we have six onsite students, plus my brother and I. I imagine we’ll get a couple more before all is settled, but still– a small group. It’ll be like a team of commandos.

If you want to work shoulder to shoulder with us, getting on-the-job test coaching during five long days, sign up soon. The early bird discount runs out in a few days.

As I was having dinner with Noah Höjeberg tonight, he said an interesting thing. “Some people think testing is mechanical, and that’s bad enough. But a lot of people seem to think the alternative to mechanical is magical.”

(Noah is the new test strategist at Scila AB, in Stockholm. Interesting guy. I’ve played a lot of testing dice with him, in the past. I meant to do the Art Show game with him, too, but we got so much into our conversation that I completely forgot.)

Mechanical and magical are false opposites. In Rapid Testing, we pursue another path: Heuristical. In other words, skilled testing, achieved through systematic study and the deliberate application of heuristics. This is neither a mechanical, algorithmic process, not is it magical, mystical. We can show it, talk about it, etc. And yet it cannot be automated.

I did not invent sympathetic testing. Anyone who says I claim to have invented it will have only read the title of this post, but nothing further. Now you know.

I may have been the first in my circle to recognize one specific benefit of sympathetic testing. But if so, that is a minor technical point. Because I know that my recognition came directly from a point of view championed by Cem Kaner. From where I stand, it came largely from him.

Okay, James, this is a bit confusing. Why are you talking about this, today?

Ajay Balamurugadas came to me today asking if I came up with the idea of “sympathetic testing.” This is an idea he’s found in Mike Kelly’s work, which derives partly from my work (on tours). It’s also in the section of Lessons Learned in Software Testing that I wrote (and which was edited by Cem Kaner and Bret Pettichord). He had already asked Cem Kaner, who said he got it from my brother [correction: Ajay tells me he saw Cem say that on a video]. But if so, I know exactly where Jon got it: from me, during a project we worked on around 1997. That’s where he was working for me on a court case where I had to do an analysis of “good enough” quality, using a method strongly influenced by Cem’s thinking about cognitive biases.

I’m talking about this today because the provenance of ideas can be confusing when people work closely together. Who cares who invented it, or named it? You might be surprised. Credit is important to people who live by their reputations in a world of ideas. Reputation is money, to put it bluntly. To achieve high income and control over your work as a tester, one way– and the only way I know, actually– is to build your public reputation. Moreover, a lot of our motivation is the respect of our peers. Therefore, this matters.

This isn’t really about “sympathetic testing”, then. Still, what IS sympathetic testing?

Sympathetic testing, as I think of it, is sometimes slightly confused with the closely related ideas of “happy path” or “positive testing.” Sympathetic testing means testing while affirming (rather than challenging) each assumption, resource, or service. But if that’s all it meant, then it would be the same as positive testing. For me, sympathetic testing means more. It means asking “what is wonderful about this product?” rather than “what is broken about this product?” I can do positive testing all day and still be focused on bugs. With sympathetic testing, I may find bugs, but that is not my focus or purpose– my main purpose is learning; building a rich model in my mind.

The insight I had, when working with my brother (way back before he was a recognized test expert in his own right) was that sympathetic testing made me better at unsympathetic testing. Searching happily prepares me to search aggressively.

Okay, then what’s really troubling you?

What bugs me is that Cem deserves more credit for sympathetic testing. But if he claims it himself, he probably thinks he will anger me, and if he gives me full credit, it would anger him: because Cem and I aren’t on speaking terms, at the moment. (I hope that’s temporary.)

I’m hereby offering him credit. Cem and I collaborated for roughly 16 years. We had probably hundreds of deep conversations about testing. Among those conversations, he lectured me on mental models and biases. We once had a fairly bitter argument about what a model is. He won that argument, thankfully, and I have been comfortable with the outcome ever since. He introduced me to Cognitive Science and pushed me deeper into Epistemology. It was that sensibility that led to me discovering a lot of things: among them the power of sympathetic testing. I’m certain that the first person I ran to with my discovery was Cem, but not only that, this is exactly the sort of thing that he was famous for encouraging: to look at things in a sympathetic way. I can’t remember the conversation itself or what he said. I bet he said something like “That’s just what I’ve been trying to tell you!”

Therefore, “sympathetic testing” is part of our joint work. Between about 1997 and 2007, it’s a good bet that I didn’t have a significant thought about testing which wasn’t also sifted and tested by Cem. We didn’t always agree, but we were extremely close. It’s for that reason that he put my name on his BBST class, even though I didn’t do any direct work on it. He was recognizing my influence on him. I’m doing that now.

I’m my own man. I have reinvented testing for myself (just as I recommend that others do). And yet a few people have deeply influenced me in that process. The three people who have had the most influence on my are my father, Jerry Weinberg, and Cem Kaner. I sometimes joke that I can sell out and offer baloney certifications to gullible testers, just like the ISTQB does– but only after those three men have died and I’m no longer seeking their respect.

[Postscript: Why did I title this piece "How I Invented Sympathetic Testing"? Because it was the best way I could think of to attract the attention of people who might be upset that I would make such a claim. Then maybe they would read this post, and feel a lot better.]

I’ve wrapped up my first visit to Switzerland!

I came to speak at the Swiss Testing Day, which attracted a lot of testers. 800 of them, I’m told. It was a very professional operation.

For me the highlight of the visit was trading puzzles and working through Socratic examinations with various Swiss testers, principally Ilari Aegerter. Ilari (which I discovered after saying his name wrong 100 times actually rhymes with Hillary) is a man with boundless energy to solve puzzles and meet the challenges of his peers. I spent two days plus several dinners quizzing him, pressing him, sparring with him, and accepting quizzing from him.

Today we worked in the ski and cheese town of Engleberg. After getting my need to learn about Swiss cheese out of the way (what we call Swiss cheese in the USA is really “Emmentaler”), we settled in the lobby of the Terrace Hotel with the majestic Alps hovering over us.

Ilari is serious about becoming a testing expert, as well as an expert testing coach. So, we spent all day combing through transcripts of his test coaching sessions, identifying and naming his coaching patterns. We’ll consult with Anne-Marie Charrett and Michael Bolton next, to see how our patterns square with them.

He’s relatively new to test coaching. We agreed he needs to work on his patience. Coaching is a lot like fishing. In a Socratic dialog, you have to set the hook in the student and patiently reel him in.

Otherwise, the transcripts looked good to me.

Yesterday I accepted a challenge from Ilari to describe a picture as completely as possible. It was a lot of fun, so I passed it on to Anne-Marie today, who used a mind-map strategy. Now we have a new testing challenge we can give to willing students. Ilari seems to have set himself the task of becoming Context-Driven testing’s best expert on observation skills and dynamics. He has a lot more reading to do (Metzger’s Laws of Seeing, for instance; and The Science of Describing), but he’s on his way.

Tonight we worked through key bits of a book about validity and reliability in qualitative research, and closed the the evening with several lateral thinking puzzles and two math puzzles.

We’re both kind of tired now. The good kind of tired.

This is my idea of fun.

I staggered out of the Cambridge Press bookstore a bit dazed, today, having gorged on 21 books. [Addendum: I mean by this that I browsed them, purchased them, and had them shipped home.] If you want to know what a Context-Driven tester reads, here it is:

• A First Course in Statistical Programming with R
• Wisdom: Its Nature, Origins, and Development
• Learning and Expanding with Activity Theory
• Sequential Analysis and Observational Methods for the Behavioral Sciences
• Observing Interaction: An Introduction to Sequential Analysis
• Human Error
• Combinatorics: A Problem Oriented Approach (Mathematical Association of America Textbooks)
• A Mathematician Comes of Age (Spectrum)
• The Golem at Large: What You Should Know about Technology (Canto)
• The Golem: What You Should Know about Science (Canto)
• A Practical Introduction to Denotational Semantics (Cambridge Computer Science Texts)
• Dialogic Inquiry: Towards a Socio-cultural Practice and Theory of Education (Learning in Doing: Social, Cognitive and Computational Perspectives)
• From Teams to Knots: Activity-Theoretical Studies of Collaboration and Learning at Work (Learning in Doing: Social, Cognitive and Computational Perspectives)
• Nuts and bolts for the social sciences
• How to Fold It: The Mathematics of Linkages, Origami and Polyhedra
• The Cambridge Handbook of Thinking and Reasoning (Cambridge Handbooks in Psychology)
• The Manuscript of Great Expectations: From the Townshend Collection, Wisbech (Cambridge Library Collection – Literary Studies)
• The Cognitive Basis of Science
• Satisficing and Maximizing: Moral Theorists on Practical Reason
• The Cambridge Handbook of Situated Cognition (Cambridge Handbooks in Psychology)
• Risk Communication: A Handbook for Communicating Environmental, Safety, and Health Risks

One of the challenges I have for the ISTQB proponents is “What do you read?” You see it’s a trap. If they tell me they read widely, deeply and liberally, I contrast that with the intellectual desert that is the ISTQB Syllabus and ask them why there is such a disconnect between their education and their professional claims. And if they read narrowly, well, there you go.

If you want to be an excellent tester, you need a good education. You didn’t get that in school (or if you’re in school, you’re not getting it), so you need to do something like what I do: scout for fabulous and offbeat books about all the matters of great testing– and testing touches EVERYTHING!

[addendum: If you are not familiar with my distaste for institutional education, before picking a fight with me, go see my book, Secrets of a Buccaneer-Scholar. I spent 26 years doing the research by which I assert that school, although not always destructive and occasionally helpful, is certainly not necessary if you want to live a successful intellectual life. Each day of my life is another data point about how wrong were the teachers who told me I would not be successful without submitting to "the game" of school they desired me to play.]

Most of the books on my list are self-explanatory. One in particular may seem strange: the manuscript of Great Expectations. I picked that one up because the photographic images of Dickens’ original manuscript is a beautiful example of how messy the creative process is. Imagine trying to put metrics on the process of writing that, with all its crossouts and insertions.Writing is exploratory. Just. Like. Testing.

Cem Kaner, who controls www.context-driven-testing.com, has announced an interesting change in his view of the Context-Driven School. He says he prefers to think of it in terms of the Context-Driven approach, not a school of thought. This is a significant change from his original view, which was that CDT is a different paradigm.

That means I’m the last of the founders of the Context-Driven School, as such, who remain true to the original vision. I will bear its torch along with any fellow travelers who wish to pursue a similar program.

Polarization? No. Paradigm!

One of the things that concerns Cem is the polarization of the craft. He doesn’t like it, anymore. I suppose he wants more listening to people who have different views about whether there are best practices or not. To me, that’s unwise. It empties the concept of much of its power. And frankly, it makes a mockery of what we have stood for. To me, that would be like a Newtonian physicist in the 1690′s wistfully wishing to “share ideas” with the Aristotelians. There’s no point. The Aristotelians were on a completely different path.

For me, Context-Driven thinking is delightfully about listening to people and talking to people about practices and dynamics of software testing. But this must happen within the humanist framework that we laid out in the seven principles of the Context-Driven school. That’s our world.

Polarization is beside the point. Polarization is a natural consequence of the fact that our world view is simply different. We are a different paradigm. Our paradigm cannot be explained or contained by any other testing paradigm, such as the Factory School, or the Analytical School. We must have the stomach to keep moving along with our program regardless of the huddled masses who Don’t Get It.

Why Is This Division Happening Now?

Cem’s change of position is happening partly because, after 16 years, he and I are no longer collaborators. Due to a simmering personal dispute (nothing to do with testing) that blew up last year, we no longer can stand to be in the same room with each other. Alas, I don’t think this will change. What that means, professionally, is that the conversations that we once had– the passionate arguments– which led to mutual accommodations and syntheses, no longer happen. This is too bad, because the Factory schoolers, who greatly outnumber us, will make good rhetoric out of any appearance of confusion between Cem and I about our visions of testing.

Meanwhile, I will say this about Cem: He’s a great man. His contributions to testing have been enormous. I disagree with him on some aspects of testing, but by and large he does great work. I’m sure if he weren’t so furious with me and I were able to talk to him without feeling an overpowering urge to kick holes in walls (I mean that literally), we would still be able to develop testing ideas together. However, I trust that whatever he does will be worth looking at. And I do have many other bright collaborators, so I’m going to be fine.

The Context-Driven School continues, because I, and those like me, are compelled to pursue excellence wherever it leads us, even if that means breaking with “conventional” software testing thinkers. I wish Cem luck as he consorts with those guys, but I fear his time will be, for the most part, wasted.