Last week, I had to cancel a class and rush home to take care of my wife, who had become unable to eat or drink anything. Turns out she needed emergency surgery to repair a blocked intestine. She’s recovering fine and I’m writing this from the chair next to her hospital bed, where I sleep and try to gain husband points by doing her bidding.

Thankfully it wasn’t cancer, but rather a condition called endometriosis. Apparently, she’s had it for years and didn’t know it. It’s kind of like having a strange recurring bug that suddenly gets explained. Lots of things make sense now that didn’t before. For instance, she has suffered chronic fatigue through most of her life, but when she was pregnant, 14 years ago, she had tremendous energy. Most women are tired during that time, but Lenore was voraciously reading and attending lectures and classes. It turns out that the endometriosis probably was the cause both of the fatigue and the cause of the energy.

Anyway, Lenore is my office manager and dispatcher, so in light of these developments, we’ve decided to cancel all my travel for the rest of the year. (So, no Eurostar. The certification fanatics will have to get someone else to wake them from their delusions.) Starting in January, I was planning to devote several months just to writing my book on self-education and finding a publisher for it (see an old draft at http://www.satisfice.com/hils.pdf). I also have a book on exploratory testing in the works. Now, that writing will begin right away.

If anyone has consulting they’d like me to do that does not require travel outside of the Seattle area, I’m willing to consider that. Otherwise, I have to write like crazy and see if I can make it pay off.

(And if you ever need major surgery, I can recommend highly the Swedish Medical Center in downtown Seattle. These people are fabulous. The customer service is much better than I expected.)

I’ve been reading a bit about the Technological Singularity. It’s an interesting and chilling idea conceived by people who aren’t testers. It goes like this: the progress of technology is increasing exponentially. Eventually the A.I. technology will exist that will be capable of surpassing human intelligence and increasing its own intelligence. At that point, called the Singularity, the future will not need us… Transhumanity will be born… A new era of evolution will begin.

I think a tester was not involved in this particular project plan. For one thing, we aren’t even able to define intelligence, except as the ability to perform rather narrow and banal tasks super-fast, so how do we get from there to something human-like? It seems to me that the efforts to create machines that will fool humans into believing that they are smart are equivalent to carving a Ferrari out of wax. Sure you could fool someone, but it’s still not a Ferrari. Wishing and believing doesn’t make it a Ferrari.

Because we know how a Ferrari works, it’s easy to understand that a wax Ferrari is very different from a real one. Since we don’t know what intelligence really is, even smart people easily will confuse wax intelligence for real intelligence. In testing terms, however, I have to ask “What are the features of artificial intelligence? How would you test them? How would you know they are reliable? And most importantly, how would you know that human intelligence doesn’t possess secret and subtle features that have not yet been identified?” Being beaten in chess by a chess computer is no evidence that such a computer can help you with your taxes, or advise you on your troubles with girls. Impressive feats of “intelligence” simply do not encompass intelligence in all the forms that we routinely experience it.

The Google Grid

One example is the so-called Google Grid. I saw a video, the other day, called Epic 2014. It’s about the rise of a collection of tools from Google that create an artificial mass intelligence. One of the features of this fantasy is an “algorithm” that automatically writes news stories by cobbling pieces from other news stories. The problem with that idea is that it seems to know nothing about writing. Writing is not merely text manipulation. Writing is not snipping and remixing. Writing requires modeling a world, modeling a reader’s world, conceiving of a communication goal, and finding a solution to achieve that goal. To write is to express a point of view. What the creators of Epic 2014 seemed to be imagining is a system capable of really really bad writing. We already have that. It’s called Racter. It came out years ago. The Google people are thinking of creating a better Racter, essentially. The chilling thing about that is that it will fool a lot of people, whose lives will be a little less rich for it.

I think the only way we can get to an interesting artificial intelligence is to create conditions for certain interesting phenomena of intelligence to emerge and self-organize in some sort of highly connectionist networked soup of neuron-like agents. We won’t know if it really is “human-like”, except perhaps after a long period of testing, but growing it will have to be a delicate and buggy process, for the same reason that complex software development is complex and buggy. Just like Hal in 2001, maybe it’s really smart, or maybe it’s really crazy and tells lies. Call in the testers, please.

(When Hal claimed in the movie that no 9000 series computers had ever made an error, I was ready to reboot him right then.)

No, you say? You will assemble the intelligence out of trillions of identical simple components and let nature and data stimulation build the intelligence automatically? Well, that’s how evolution works, and look how buggy THAT is! Look how long it takes. Look at how narrow the intelligences are that it has created. And if we turn a narrow and simplistic intelligence to the task of redesigning itself, why suppose that it is more likely to do a good job than a terrible job?

Although humans have written programs, no program yet has written a human. There’s a reason for that. Humans are oodles more sophisticated than programs. So, the master program that threatens to take over humanity would require an even more masterful program to debug itself with. But there can’t be one, because THAT program would require a program to debug itself… and so on.

The Complexity Barrier

So, I predict that the singularity will be drowned and defeated by what might be called the Complexity Barrier. The more complex the technology, the more prone to breakdown. In fact much of the “progress” of technology seems to be accompanied by a process of training humans to accept increasingly fragile technology. I predict that we will discover that the amount of energy and resources needed to surmount the complexity barrier will approach infinity.

In the future, technology will be like weather. We will be able to predict it somewhat, but things will go mysteriously wrong on a regular basis. Things fall apart; the CPU will not hold.

Until I see a workable test plan for the Singularity, I can’t take it seriously.

I just read this on another blog:

“Regression testing is usually seen as the poorer cousin of ‘proper’ domain-abstracted assertion-based testing. Often rightly so!”

In twenty years of doing testing, managing testing, attending many many conferences and reading many papers and books on testing, I have not heard of “domain-abstracted assertion-based testing”.

I don’t know what assertion-based testing is. So if it’s usually associated with regression testing, then that must be true of a community other than the ones I’ve encountered.

I don’t know what it means to be domain-abstracted. Again, perhaps some community of testers obscure to me uses that term.

Moreover, I find that when I imagine what those things might be– and I thought I was good at imagining things– the ideas I come up with don’t seem anything like regression testing.

I often exhort testing students that it’s okay to make up your own terminology. I am not in favor of a standardized way of speaking except within specific communities, companies, or projects. But this case reminds me to add this caveat: Remember friends, when you invent words and terms, it sure helps if you align them with some aspect of some corpus of existing vocabulary, so that the rest of us have a hope of figuring out what you mean.

Regression testing is a very widely used term in testing. I’m aware of no testing community that does not use that particular term. I googled “domain-abstracted testing” and found zero hits. However I googled “assertion-based testing” and found about 500 hits, some of them interesting.

I would have thought “assertion-based testing” meant embedding ASSERT calls in code. In other words, built-in tests. That would help detect regression-related bugs, but also bugs that aren’t regression-related, so I wouldn’t call it form of regression testing per se. As I google the term, it seems that it is used in various ways.

Some terms used commonly in testing I like to call “danger words”, by which I mean that using them without defining them will confuse people in most contexts that occur to me. It looks like “assertion” is an example of a danger word.

Still, I don’t really mind all this. I’d rather learn from how people baffle me with their words than attempt to stop all innovation and free thinking by legislating a lexicon.

This is my new office on Orcas Island. The rest of the house is piled high with boxes, but at least this part is functional. The body of water outside is East Sound, which is about a mile wide. The west side of the island rises up on the other side.

It’s peaceful here, except for the seaplanes that take off under my window, four times a day, and the occasional speedboat. A raccoon just walked across my deck, a minute ago. A pack of ten mule deer hover like paparazzi outside our front gate because the previous owner fed them honey oats.

We have DSL now, and it looks likely that I will be able to get 2.5 megabit symmetrical wifi beamed from an access point across the East Sound. If that comes through, I will be able to conduct some great online training, next year.

My plan (starting January) is to travel a lot less, and do a lot more writing and online training.

I just finished a 10 day drive across the U.S. with my son. We just moved from Virginia to Orcas Island, Washington. My Internet and telephone is not yet set up, and we are madly trying to direct the movers and such. If you have emailed me, this is why I’m so slow in responding.

I anticipate it will be a couple weeks before I have a hope of catching up.

I found this gem in the FDA Glossary of Computerized System and Software Development Terminology:

“unambiguous. (1) Not having two or more possible meanings. (2) Not susceptible to different interpretations. (3) Not obscure, not vague. (4) Clear, definite, certain.”

Strangely, their glossary does not define ironic.

Have you ever done something manually? Have you ever tried to automate it? Did you successfully automate what you were doing?

If you answered yes for any of these questions, then I’m afraid I’m being too vague– because at least three very different kinds of things are tangled together in a simple answer. In any activity done by a human, there is the human aspect (not practically mechanizable), a physical aspect involving translation or transformation of matter and energy (mechanizable in principle), and a problem-solving aspect (sometimes transformed by mechanization, sometimes not affected). Which of those get automated when we automate? Which must remain “manual”? What problems are solved and what new problems are created?

My business is software testing. I have heard many people say they are in my business, too. Sometimes, when these people talk about automating tests, I think they probably aren’t in my business, after all. They couldn’t be, because what I think I’m doing is very hard to automate in any meaningful way. So I wonder… what the heck are they automating?

Anyway, this confusion is why I am applying a new term to processes: sapience. Sapience is defined in the dictionary as “wisdom; sagacity.” I want to suggest a particular connotation. A sapient process is any process that relies on skilled humans.

A sapient process might gain from automating some or all of the purely material aspects of it, but any human aspect that is replaced or displaced by machinery results would in an impairment of that process. This is a matter of definition, not fact. I’m defining sapient processes as that which we do not know how to automate without dumbing down the process. It will either be slightly less intelligent or amazingly less intelligent.

The question a test automator should ask right away is whether testing is a sapient process. I think good testing is very sapient. That’s how I experience it and teach it. My brain is constantly turned on while testing. Show me someone who’s brain is not engaged, and I’ll show you a poor tester or a badly designed test.

Is digging a hole a sapient process? It might be. Consider an archeological dig. There’s no simple algorithm or tool that will automatically excavate a valuable historical site while we go and watch TV.

The purpose of this terminology is to help make it clearer what kinds of processes we are referring to. From now on, I will try not to use the term “manual testing”, except in quotes. I will practice saying sapient testing. I believe that nearly all “manual testing” is better being called sapient. I’m not carrying rocks out here, I’m penetrating the illusions surrounding a software product. Machines can’t do that.

To fully automate a sapient test activity would require us to find an alternate version of that activity which solved the same problems (or more) and that had no human element in it. If you think you know where that has been done, please tell me about it.

This posting by Corey Goldberg illustrates an interesting and all too common kind of confusion people get into when discussing methods and practices. It’s worth pondering.

On SQAForums, someone stated:

“ISEB defines automated tested as useful only in mature testing environments and where functionality is not changing i.e. at regression testing.”

to which Corey replied:

“…and ISEB would be completely wrong on that point. web services testing should be fully automated, as there is no UI, just an API.”

Let’s analyze these statements. The first writer seems to be under the sway of ISEB, which immediately induces a heavy sigh in the pit of my soul.

(There are now thousands of people who might be called “certification zombies” lurching around in an ISEB or ISTQB-induced fog, trying to apply what they learned in a few days of memorizing to the complex reality of testing.)

When the first writer says that ISEB “defines” automation as useful only in a certain context, that’s a perfect example of the inability to separate context and method. To think clearly about methodology, you must be able to sift these things apart. Best practice thinking can’t help you do this, and in fact discourages you from trying.

I don’t know if ISEB actually defines or discusses test automation in that way, but if it does, I can tell you what ISEB is probably thinking.

(BTW, one of the big problems with certification programs is the depersonalization of convictions. I say “ISEB” when what I want to say is Dorothy Graham or one of those people who support and edit the ISEB syllabus. You can’t argue with a document. Only people can have a point of view. To argue with ISEB itself is to argue with an anonymous sock puppet. But that’s the way they want it. Certificationists quite purposefully create a bureaucratic buffer of paper between themselves and any dissenters. To pick someone whom I believe advocates the ISEB way, I will choose Dorothy Graham.)

If Dot advocates that belief, then she is probably thinking about GUI-level automation of some aspects of test execution; a set of detailed scripted actions programmed into a software agent to exercise a system under test. If so then it is indeed likely that modifying the system under test in certain ways will break the test automation. This often leads to a situation where you are constantly trying to fix the automation instead of enjoying the benefits of it. This is especially a problem when the testing is happening via a GUI, because little changes that don’t bother a human will instantly disable a script.

So, even though the first writer appears to be reading off the ISEB script, there is some validity to his claim, in some context.

Now look at Corey’s reply. Corey is not under the sway of ISEB, but I worry that he may be under the sway of a typical affliction common among programmers who talk about testing: the reification fallacy. This is the tendency to think of an abstraction or an emergent process as if it were a fixed concrete thing. Hence if a programmer sees me punch a few keys in the course of my testing, and writes a program that punches those same keys in the same order, he might announce that he as “automated the test”, as if the test were nothing more than a pattern of input and output. Certainly, it is possible to automate some aspects of testing, but the aspect of it that requires human reflection cannot be automated. In fact, it can’t even be precisely duplicated by another human. It is an emergent phenomenon.

(Some would say that I am splitting hairs too finely, and that imprecise duplication may be close enough. I agree that it may be close enough in certain contexts. What I caution against is taking the attitude that most of what is valuable about testing, most of the time, is easy to automate. When I have seen that attitude in practice, the resulting automation has generally been too expensive and too shallow. Rich, interesting, cost-effective test automation, in my experience, is a constructive partnership between human thinkers and their tools. I believe, based on my knowledge of Corey, that he actually is interacting constructively with his tools. But in this case, he’s not talking that way.)

What Corey can do is use tools to interact with a system under test. He uses his un-automatable human mind to program those tools to provide certain input and look for certain output. His tools will be able to reveal certain bugs. His tools in conjunction with un-automatable human assistance during and after execution and un-automatable human assistance to re-program the tests as needed will reveal many more bugs.

The reification fallacy leads to certain absurdities when you consider different frames of reference. Corey points out that a web service has no “user interface”, and therefore is accessible only via a tool, and anything that is accessible only by a tool must therefore require “fully automated” testing. By that reasoning, we can say that all testing is always fully automated because in all cases there is some kind of hardware or software that mediates our access to the object of our test. Therefore, the fact that I am using a keyboard to type this blog posting and a screen to view it, by Cory’s logic, must be fully automated writing! I wonder what will be written next by my magic keyboard?

From one frame of reference, a web service has no user interface. From another frame of reference we can say that it does have a user interface, just not a human interface– its user is another program. How we test such a thing is to write or employ a program that does have a human interface to manipulate the web service. We can operate this interface in batch mode: write a program to submit data, run it, review the results, and re-write the program as needed. Or we can operate the interface interactively: write a program to submit data, present results, then wait for us to type in a new query.

Corey and the first writer are not in a helpful dialog, because they are talking about different things. I would tell the first writer to treat ISEB as having no authority or wisdom, and to instead learn to reason for himself. The relevant reasoning here, I think, is to wonder what kind of tool we could find or write that would allow us to interact with the web service. At the same time, we need to consider how the web service interface might change. We might stick to highly interactive testing for a while, instead of investing in a batching system with lot of automatic oracles, if we feel that the interface and functionality is changing too fast. On the other hand, one of the nice things about testing through an API is that it is often rather inexpensive to script sequences and batches and simple oracles; and consequently inexpensive to fix them when the system under test changes. I suspect that belief informed Corey’s response, although I wish he would make that belief more apparent to people who are used to thinking of testing as a human-driven process.

As a programmer, I am aware of the urge, sometimes, to say “I didn’t do it, my program did.” In testing this naturally turns into “I didn’t test that, my program I wrote to test that did.” The crucial difficulty with this way of speaking, when it comes to testing, is the way it obscures the many, many choices the programmer made while designing the program, as if the program itself made those choices, or as if there were no choices to be made. The thing is, I don’t care, for a regular program, how many other ways it could have been written, or how any other things it could have done. But these are vital concerns when the program is meant to test another program.

When I first joined ST Labs, years ago, we faced a dilemma. We had clients telling us what kind of bugs we should not report. “Don’t worry about the installer. Don’t report bugs on that. We have that covered.” No problem, dear customer, we cheerfully replied. Then after the project we would hear complaints about all the installation bugs we “missed”.

So, we developed a protocol called Mention In Passing, or “mipping”. All bugs shall be reported, without exception. Any bug that seems questionable or prohibited we will “mention in passing” in our status reports or emails. In an extreme case we mention it by voice, but I generally want to have a written record. That way we are not accused of wasting time investigating and reporting the bug formally, but we also can’t be accused of missing it entirely.

If a client tells me to stop bothering him about those bugs, even in passing, I might switch to batching them, or I might write a memo to all involved that I will henceforth not report that kind of problem. But if there is reasonable doubt in my mind that my client and I have a strong common understanding of what should and should not be reported, I simply tell them that I “mip” bugs to periodically check to see if I have accidentally misconstrued the standard for reporting, or to see if the standard has changed.

STAR East is next week, in Orlando, Fl. I will be there. I’ll be roaming the halls with my portable test coaching kit (currently being beta tested), hoping to talk to people.

I rarely go to a presentation at an exhibition conference like STAR, because I go to conferences to confer, and that’s a two-way process. But I appreciate being invited to a talk by a speaker who wants a critical and collegial review. (Whenever I give a talk, I invite anyone and everyone present to talk back, argue with me, attack me, etc. That’s how I earn my self-respect.)

So, mostly I’ll be in the hallways and in the lounge, sipping my ginger ale. If you want free consulting or coaching while I’m there, I don’t charge for that– actually that’s how I do my marketing. And if you want to offer me free coaching or consulting, too, be my guest.

You know what I would love? Challenge me to test something. Throw down a laptop in front of me and say “Test this! Test it now!”

I will also be at the CAST conference, in Seattle, in July, where I will be holding an open testing competition and giving my “Self-Education for Testers” tutorial. CAST is the closest thing we have to an exhibition conference for Context-Driven testers. And I’ll be at Eurostar, in Stockholm, in December, where I will be making my arguments against certification programs such as ISTQB and explaining why I am not certified.