I found this gem in the FDA Glossary of Computerized System and Software Development Terminology:

“unambiguous. (1) Not having two or more possible meanings. (2) Not susceptible to different interpretations. (3) Not obscure, not vague. (4) Clear, definite, certain.”

Strangely, their glossary does not define ironic.

Have you ever done something manually? Have you ever tried to automate it? Did you successfully automate what you were doing?

If you answered yes for any of these questions, then I’m afraid I’m being too vague– because at least three very different kinds of things are tangled together in a simple answer. In any activity done by a human, there is the human aspect (not practically mechanizable), a physical aspect involving translation or transformation of matter and energy (mechanizable in principle), and a problem-solving aspect (sometimes transformed by mechanization, sometimes not affected). Which of those get automated when we automate? Which must remain “manual”? What problems are solved and what new problems are created?

My business is software testing. I have heard many people say they are in my business, too. Sometimes, when these people talk about automating tests, I think they probably aren’t in my business, after all. They couldn’t be, because what I think I’m doing is very hard to automate in any meaningful way. So I wonder… what the heck are they automating?

Anyway, this confusion is why I am applying a new term to processes: sapience. Sapience is defined in the dictionary as “wisdom; sagacity.” I want to suggest a particular connotation. A sapient process is any process that relies on skilled humans.

A sapient process might gain from automating some or all of the purely material aspects of it, but any human aspect that is replaced or displaced by machinery results would in an impairment of that process. This is a matter of definition, not fact. I’m defining sapient processes as that which we do not know how to automate without dumbing down the process. It will either be slightly less intelligent or amazingly less intelligent.

The question a test automator should ask right away is whether testing is a sapient process. I think good testing is very sapient. That’s how I experience it and teach it. My brain is constantly turned on while testing. Show me someone who’s brain is not engaged, and I’ll show you a poor tester or a badly designed test.

Is digging a hole a sapient process? It might be. Consider an archeological dig. There’s no simple algorithm or tool that will automatically excavate a valuable historical site while we go and watch TV.

The purpose of this terminology is to help make it clearer what kinds of processes we are referring to. From now on, I will try not to use the term “manual testing”, except in quotes. I will practice saying sapient testing. I believe that nearly all “manual testing” is better being called sapient. I’m not carrying rocks out here, I’m penetrating the illusions surrounding a software product. Machines can’t do that.

To fully automate a sapient test activity would require us to find an alternate version of that activity which solved the same problems (or more) and that had no human element in it. If you think you know where that has been done, please tell me about it.

This posting by Corey Goldberg illustrates an interesting and all too common kind of confusion people get into when discussing methods and practices. It’s worth pondering.

On SQAForums, someone stated:

“ISEB defines automated tested as useful only in mature testing environments and where functionality is not changing i.e. at regression testing.”

to which Corey replied:

“…and ISEB would be completely wrong on that point. web services testing should be fully automated, as there is no UI, just an API.”

Let’s analyze these statements. The first writer seems to be under the sway of ISEB, which immediately induces a heavy sigh in the pit of my soul.

(There are now thousands of people who might be called “certification zombies” lurching around in an ISEB or ISTQB-induced fog, trying to apply what they learned in a few days of memorizing to the complex reality of testing.)

When the first writer says that ISEB “defines” automation as useful only in a certain context, that’s a perfect example of the inability to separate context and method. To think clearly about methodology, you must be able to sift these things apart. Best practice thinking can’t help you do this, and in fact discourages you from trying.

I don’t know if ISEB actually defines or discusses test automation in that way, but if it does, I can tell you what ISEB is probably thinking.

(BTW, one of the big problems with certification programs is the depersonalization of convictions. I say “ISEB” when what I want to say is Dorothy Graham or one of those people who support and edit the ISEB syllabus. You can’t argue with a document. Only people can have a point of view. To argue with ISEB itself is to argue with an anonymous sock puppet. But that’s the way they want it. Certificationists quite purposefully create a bureaucratic buffer of paper between themselves and any dissenters. To pick someone whom I believe advocates the ISEB way, I will choose Dorothy Graham.)

If Dot advocates that belief, then she is probably thinking about GUI-level automation of some aspects of test execution; a set of detailed scripted actions programmed into a software agent to exercise a system under test. If so then it is indeed likely that modifying the system under test in certain ways will break the test automation. This often leads to a situation where you are constantly trying to fix the automation instead of enjoying the benefits of it. This is especially a problem when the testing is happening via a GUI, because little changes that don’t bother a human will instantly disable a script.

So, even though the first writer appears to be reading off the ISEB script, there is some validity to his claim, in some context.

Now look at Corey’s reply. Corey is not under the sway of ISEB, but I worry that he may be under the sway of a typical affliction common among programmers who talk about testing: the reification fallacy. This is the tendency to think of an abstraction or an emergent process as if it were a fixed concrete thing. Hence if a programmer sees me punch a few keys in the course of my testing, and writes a program that punches those same keys in the same order, he might announce that he as “automated the test”, as if the test were nothing more than a pattern of input and output. Certainly, it is possible to automate some aspects of testing, but the aspect of it that requires human reflection cannot be automated. In fact, it can’t even be precisely duplicated by another human. It is an emergent phenomenon.

(Some would say that I am splitting hairs too finely, and that imprecise duplication may be close enough. I agree that it may be close enough in certain contexts. What I caution against is taking the attitude that most of what is valuable about testing, most of the time, is easy to automate. When I have seen that attitude in practice, the resulting automation has generally been too expensive and too shallow. Rich, interesting, cost-effective test automation, in my experience, is a constructive partnership between human thinkers and their tools. I believe, based on my knowledge of Corey, that he actually is interacting constructively with his tools. But in this case, he’s not talking that way.)

What Corey can do is use tools to interact with a system under test. He uses his un-automatable human mind to program those tools to provide certain input and look for certain output. His tools will be able to reveal certain bugs. His tools in conjunction with un-automatable human assistance during and after execution and un-automatable human assistance to re-program the tests as needed will reveal many more bugs.

The reification fallacy leads to certain absurdities when you consider different frames of reference. Corey points out that a web service has no “user interface”, and therefore is accessible only via a tool, and anything that is accessible only by a tool must therefore require “fully automated” testing. By that reasoning, we can say that all testing is always fully automated because in all cases there is some kind of hardware or software that mediates our access to the object of our test. Therefore, the fact that I am using a keyboard to type this blog posting and a screen to view it, by Cory’s logic, must be fully automated writing! I wonder what will be written next by my magic keyboard?

From one frame of reference, a web service has no user interface. From another frame of reference we can say that it does have a user interface, just not a human interface– its user is another program. How we test such a thing is to write or employ a program that does have a human interface to manipulate the web service. We can operate this interface in batch mode: write a program to submit data, run it, review the results, and re-write the program as needed. Or we can operate the interface interactively: write a program to submit data, present results, then wait for us to type in a new query.

Corey and the first writer are not in a helpful dialog, because they are talking about different things. I would tell the first writer to treat ISEB as having no authority or wisdom, and to instead learn to reason for himself. The relevant reasoning here, I think, is to wonder what kind of tool we could find or write that would allow us to interact with the web service. At the same time, we need to consider how the web service interface might change. We might stick to highly interactive testing for a while, instead of investing in a batching system with lot of automatic oracles, if we feel that the interface and functionality is changing too fast. On the other hand, one of the nice things about testing through an API is that it is often rather inexpensive to script sequences and batches and simple oracles; and consequently inexpensive to fix them when the system under test changes. I suspect that belief informed Corey’s response, although I wish he would make that belief more apparent to people who are used to thinking of testing as a human-driven process.

As a programmer, I am aware of the urge, sometimes, to say “I didn’t do it, my program did.” In testing this naturally turns into “I didn’t test that, my program I wrote to test that did.” The crucial difficulty with this way of speaking, when it comes to testing, is the way it obscures the many, many choices the programmer made while designing the program, as if the program itself made those choices, or as if there were no choices to be made. The thing is, I don’t care, for a regular program, how many other ways it could have been written, or how any other things it could have done. But these are vital concerns when the program is meant to test another program.

When I first joined ST Labs, years ago, we faced a dilemma. We had clients telling us what kind of bugs we should not report. “Don’t worry about the installer. Don’t report bugs on that. We have that covered.” No problem, dear customer, we cheerfully replied. Then after the project we would hear complaints about all the installation bugs we “missed”.

So, we developed a protocol called Mention In Passing, or “mipping”. All bugs shall be reported, without exception. Any bug that seems questionable or prohibited we will “mention in passing” in our status reports or emails. In an extreme case we mention it by voice, but I generally want to have a written record. That way we are not accused of wasting time investigating and reporting the bug formally, but we also can’t be accused of missing it entirely.

If a client tells me to stop bothering him about those bugs, even in passing, I might switch to batching them, or I might write a memo to all involved that I will henceforth not report that kind of problem. But if there is reasonable doubt in my mind that my client and I have a strong common understanding of what should and should not be reported, I simply tell them that I “mip” bugs to periodically check to see if I have accidentally misconstrued the standard for reporting, or to see if the standard has changed.

STAR East is next week, in Orlando, Fl. I will be there. I’ll be roaming the halls with my portable test coaching kit (currently being beta tested), hoping to talk to people.

I rarely go to a presentation at an exhibition conference like STAR, because I go to conferences to confer, and that’s a two-way process. But I appreciate being invited to a talk by a speaker who wants a critical and collegial review. (Whenever I give a talk, I invite anyone and everyone present to talk back, argue with me, attack me, etc. That’s how I earn my self-respect.)

So, mostly I’ll be in the hallways and in the lounge, sipping my ginger ale. If you want free consulting or coaching while I’m there, I don’t charge for that– actually that’s how I do my marketing. And if you want to offer me free coaching or consulting, too, be my guest.

You know what I would love? Challenge me to test something. Throw down a laptop in front of me and say “Test this! Test it now!”

I will also be at the CAST conference, in Seattle, in July, where I will be holding an open testing competition and giving my “Self-Education for Testers” tutorial. CAST is the closest thing we have to an exhibition conference for Context-Driven testers. And I’ll be at Eurostar, in Stockholm, in December, where I will be making my arguments against certification programs such as ISTQB and explaining why I am not certified.

This post was a criticism of a post by Paul Gerrard that has been withdrawn. Hence, I withdraw my criticism.

Recently, Michael Bolton and I had a conversation, which he recorded, about the deep nature and meaning of a scripted test procedure. It’s an MP3 file that is a bit more than an hour long.

If you think that scripting a test is a simple matter of “writing down what the tester should do”, then I would urge you to listen to Michael and I deconstruct a simple example of a test script.

We take the first step of a hypothetical test and analyze it. That step is:

“1. Reboot the test system.”

What does this mean? What should the tester do? Consider these questions for a moment before listening to the MP3.

It would be an unsound fancy and self-contradictory to expect that things which have never yet been done can be done except by means which have never yet been tried.

This reminds me of that “definition of insanity” which is so often attributed to Einstein. But this comes from Francis Bacon, circa 1620. In his seminal work “The New Organon”.

Bacon has some other interesting quotes for testers…

The logic now in use serves rather to fix and give stability to the errors which have their foundation in commonly received notions than to help the search after truth. So it does more harm than good.

This is how I feel about the ISTQB syllabus.

There is no soundness in our notions, whether logical or physical. Substance, Quality, Action, Passion, Essence itself, are not sound notions; much less are Heavy, Light, Dense, Rare, Moist, Dry, Generation, Corruption, Attraction, Repulsion, Element, Matter, Form, and the like; but all are fantastical and ill defined.

This is how I feel about a lot of testing terminology.

It is idle to expect any great advancement in science from the superinducing and engrafting of new things upon old. We must begin anew from the very foundations, unless we would revolve forever in a circle with mean and contemptible progress.

I feel that way, too, about patterns such as the v-model, and most of what passed for test techniques in the 80’s.

Francis Bacon was proposing a great break with the stifling Aristotelianism of his day (the certification craze of the middle ages), and sought a new foundation for science. Bacon thereafter became a godfather of the enlightenment, helping to create the modern world.

We’re way beyond Bacon, now. Still, I’m attracted to his sentiment that what passed for good scientific work in his time was actually nothing but uncritical folklore. In our time, in our little field, we need a similar re-invention of the craft, a New Organon of testing.

People have been asking me why I haven’t been blogging. Well, here it is. I arrived in London, today. I’m going to Scotland next week. I just came from Sweden. I arrived in Sweden from Australia, by way of New Zealand. This is the longest and most intense road trip I’ve done since I started my company, eight years ago. I’m tired! No, here’s a better word: I feel dazed. If I’m not traveling, then I’m talking. My life for the last five weeks has been one conversation and demonstration after another, every day, even weekends.

I’ve taught at specific companies as well as doing public classes. I did a conference in Sweden and a SIG meeting in Christchurch. I consulted for two medical device companies that are using exploratory testing, and did a rapid testing clinic for an electronics company. I’m hoping they will let me write about what happened there. It was wonderful for me to see people demonstrating Rapid Testing on the job.

Although I have not been blogging, I have been getting lots of ideas. So much to talk about: new heuristics, the “mipping” of bugs, the importance of dynamic focus, latent testing skills, how to hold a testing clinic, doing standing state testing, verbal self-defense for testers, and my portable test coaching kit. Lots of stuff.

Adelaide and Christchurch were wonderful as usual. But I’m really coming to love Sweden and the Swedish testing spirit. To be sure, I’ve only seen a bit of it. But that bit is impressive. I make a point of meeting with Tobbe Ryber each time I visit. He taught me a game we call “plenty questions” that I will be blogging about, at some point. Pablo Garcia charmed me with wine, cheese, and his cheerfully incisive outlook on the testing world. Anders Claesson revealed himself to be a philosophical soul. Imagine someone with the voice and look of an accountant, but the subtle mind and resources of Gandalf the Grey– if Gandalf had been a tester.

Context-driven testing has been a predominantly American community. Some of it is also happening in England. But Sweden appears to be the emerging hotbed. Someone told me that the Swedish management culture is fond of diversity and individual initiative combined with teamwork. Maybe that explains it.

Michael Hunter interviewed me for DDJ. See the interview here.

Michael’s a cool guy. He just took the Rapid Software Testing class. It’s exciting to me when a Microsoftie takes the class, considering how hostile Microsoft has become to the idea that testing could be a skilled activity, distinct from programming.