What Could Kill Testing?

(I wrote this several years ago with Michael Bolton, but never got around to publishing it… UPDATE: Oh! I did publish this as an editorial in Tea Time for Testers. Well, anyway this is an update of it…)

Although Rome wasn’t built in a day, it took six days to accidentally burn down in 64AD. It was rebuilt to be a bit more fireproof. And when the burning of the Iroquois Theater in Chicago killed 602 people in 1903, the fire code for theaters improved in 1904. The Triangle Waistcoat Factory fire in New York City (146 dead) led to the founding of the New York City Bureau of Fire Protection, and the National Fire Protection Association now maintains several hundred separate codes– many of them inspired directly by specific tragic fires. People learn from disasters.

This is also why we have testers. Software disasters happened and people learned. Those specific people became more careful, dedicated more energy to quality assurance (including testing), and there were fewer disasters. But, unlike fire codes, devotion to QA is generally not a matter of law. If an organization hasn’t had a disaster in a while, their practices get steadily riskier (partly because younger and more innocent people replace the experienced ones). This is a normal Darwinian cycle.

So it’s not entirely surprising that at the STARWest conference, in 2011, James Whittaker (then at Google) announced that testing is “dead.” What? Testing is dead?! He seemed to be saying that testers are no longer needed in a world with automated checks and automatic updates. But Whittaker was not a professional tester. So, imagine a cabinet factory industrialist, never himself having built a cabinet, announcing the death of skilled carpentry. That’s what it sounded like to me.

As if the Fates had overheard him and been offended, a few weeks later a bunch of Google bugs made news: an article appeared on CNN.com with the lamentable title “The week Google really messed up.” A couple months after that, Google Wallet was discovered to have a serious security problem affecting all users. More bad publicity.

What does that mean for the testing field? After all, no testing process is guaranteed to save us from all bugs. Meanwhile, other processes can find bugs, too, or prevent them. Amateurs and part-timers can find bugs. Programmers can test their own code. Just because Google gets embarrassed now and then by bad software doesn’t automatically mean they should hire more testers. Maybe instead they should hire better programmers, or train them better.

Well, one thing seems obvious: bragging about how you don’t value testing is strange when you also expect forgiveness from your customers and (increasingly) world governments when you hurt society with your products.

What Would Kill Testing?

Testing is not dead. Testing won’t be dead. And anywhere testing seems to die it will be reborn, phoenix-like, not exactly from it’s own ashes, but rather from the consequences of its death. Still, it can be a good exercise to think about what might cause the death of testing, even in a temporary way. Michael Bolton and I sat down recently to brainstorm on that. Here’s what we came up with:

1. Testing may die if you start using the word “testing” to mean checking. One of Michael’s contributions to the craft was to suggest a sharp distinction between testing and mere output checking. To test is to question the product so as to evaluate it. Testing is an open-ended investigation that cannot be automated. To check, however, is to gather specific information and analyze it in a manner that could, in principle, be automated. In the parlance of philosophers, checking is a mimeomorphic activity; testing is polimorphic. Some people, mainly programmers who don’t study testing much, are strongly attached to automation. In pursuing their vision of applying tools to testing, they inadvertently dumb testing down. They do with tools what tools can do. They run many checks. Testing for them becomes little more than a command-line switch on the compiler (“-t for test”, or -q for “put the quality in”). And such checks are capable of finding bugs, just not nearly the breadth and depth and variety of bugs that a skilled human can, especially if that human also uses tools in a supporting role.

Mistaking testing for checking can kill testing, in a sense, by co-opting testing practice. Testing, as Michael and I see it, would still exist, of course. But it would be relegated to rhetorical shadows.

What I mean by that is that few people would systematically learn how to test, anymore, until we came up with new words that referred to what the term “testing” once meant. To systematically learn a technical subject, you must be able to talk about it. If all your words about testing refer to shallow and mechanical processes, the deep and skilled stuff is not a part of your world.

2. Testing may die if the value of products becomes irrelevant. It dies when we don’t care about the quality of software or the people who need it. By the same token, if we always trusted the water we drank, or the meat we bought at the store, then water testing and food hygiene standards would be irrelevant. If we didn’t mind the occasional deadly fire, we’d happily see a show down at the ol’ Iroquois theater.

There really is a problem in our industry with the erosion of the expectation that anything will work reliably, ever. I was trapped outside my house, in the cold, recently, and found that my Android phone would not make any calls on the cell network. I rebooted the phone (that takes a few minutes). Still no joy. I connected to WiFi and tried to call that way but got a strange error about not being registered. I had made calls through
WiFi before from my house, so I knew it could work. Finally I started Skype and IM’d my son (this was through WiFi , so why didn’t the phone calls work?). This thing is supposed to be a phone. I’m annoyed, but not surprised.

Google probably thinks I’m not going to give up my phone just because of a few glitches. This creates an opportunity for competitors to come in with a better product that kicks them out of the market— but hey— I worked for Apple, years ago, and when you’re inside a big company like that, you don’t really care. You think success is your birthright.

(Since I first wrote this I switched to an iPhone, which has been somewhat better.)

3. Testing may die if the quality of testing work is chronically poor.  Unfortunately, the death of testing can be a self-fulfilling prophecy. People most likely to believe that testing is dead are — like the folks at Google —unlikely to devote themselves to the study of it. They simply don’t know how to test, or perhaps don’t care. It’s only a matter of time before management wonders why they have testers at all.

The antidote for that is a high standard of personal excellence. This is what the Context-Driven testing community stands for. We are doing our best to win over the rest of the testing world by being good role models.

4. Testing may die if all the users in the world were early adopter technocrats.  Let’s pretend that all the people in the world who use computers or rely on them in some way are highly technical and tolerant of problems in the products they use. Then the need for testing would dramatically fall. Sure, they want great quality, but if they don’t get it, they understand. For minor glitches, they will have the patience to find a work around.

That may be more true for the perpetual beta products that Google famously offers, but everyone on Earth depends on computers in some way, even if they’ve never seen one. And a tiny minority of those people will experimentally download a tool like Google Earth, as I have, and then spend an hour re-configuring it so that it will actually run.

5. Testing may die by suffocation. If testers are forced to channel all their ideas through a limiting set of artifacts or tools, their productivity may collapse. I’m talking about elaborate test plan templates, test script templates and test management tools, Cucumber “executable specifications” or other automation tools that require the tester to express himself only in stilted and limited ways.

That will kill testing because it turns testers into tool jockeys, whose standard of success is the weight of paper or volume of data or lines of code — none of which has much to do with testing.  Tools can be marvelously helpful in moderation, but the excellent tester will resist obsessions with tools, documents, or anything that systematically impedes the variety and profundity of his work.

6. Testing may die if technology stops changing. Testing is questioning the product. There isn’t much call to question a product that stays the same, especially if it operates in an environment and for a user base that also doesn’t change. The ambition to innovate is what invigorates the need for testers. Take away that ambition and we all will have to get jobs in comic book stores.

7. Testing may die by starvation.  When companies reward people who take unknown risks, but not people who discover what those risks actually are, testing is not being nourished. If the craft becomes uninviting to smart, talented, motivated people because you’ve turned it into a boring, uninteresting activity: that also will starve testing. The only people left would be the ones who are too frightened or lazy to leave. The reputation of testing would become steadily worse.

Michael and I teach Rapid Software Testing, which is like a martial art of testing. It’s exciting. We are trying to show people that their jobs don’t have to suck. We feed the testers.