(I wrote this several years ago with Michael Bolton, but never got around to publishing it… UPDATE: Oh! I did publish this as an editorial in Tea Time for Testers. Well, anyway this is an update of it…)
Although Rome wasn’t built in a day, it took six days to accidentally burn down in 64AD. It was rebuilt to be a bit more fireproof. And when the burning of the Iroquois Theater in Chicago killed 602 people in 1903, the fire code for theaters improved in 1904. The Triangle Waistcoat Factory fire in New York City (146 dead) led to the founding of the New York City Bureau of Fire Protection, and the National Fire Protection Association now maintains several hundred separate codes– many of them inspired directly by specific tragic fires. People learn from disasters.
This is also why we have testers. Software disasters happened and people learned. Those specific people became more careful, dedicated more energy to quality assurance (including testing), and there were fewer disasters. But, unlike fire codes, devotion to QA is generally not a matter of law. If an organization hasn’t had a disaster in a while, their practices get steadily riskier (partly because younger and more innocent people replace the experienced ones). This is a normal Darwinian cycle.
So it’s not entirely surprising that at the STARWest conference, in 2011, James Whittaker (then at Google) announced that testing is “dead.” What? Testing is dead?! He seemed to be saying that testers are no longer needed in a world with automated checks and automatic updates. But Whittaker was not a professional tester. So, imagine a cabinet factory industrialist, never himself having built a cabinet, announcing the death of skilled carpentry. That’s what it sounded like to me.
As if the Fates had overheard him and been offended, a few weeks later a bunch of Google bugs made news: an article appeared on CNN.com with the lamentable title “The week Google really messed up.” A couple months after that, Google Wallet was discovered to have a serious security problem affecting all users. More bad publicity.
What does that mean for the testing field? After all, no testing process is guaranteed to save us from all bugs. Meanwhile, other processes can find bugs, too, or prevent them. Amateurs and part-timers can find bugs. Programmers can test their own code. Just because Google gets embarrassed now and then by bad software doesn’t automatically mean they should hire more testers. Maybe instead they should hire better programmers, or train them better.
Well, one thing seems obvious: bragging about how you don’t value testing is strange when you also expect forgiveness from your customers and (increasingly) world governments when you hurt society with your products.
What Would Kill Testing?
Testing is not dead. Testing won’t be dead. And anywhere testing seems to die it will be reborn, phoenix-like, not exactly from it’s own ashes, but rather from the consequences of its death. Still, it can be a good exercise to think about what might cause the death of testing, even in a temporary way. Michael Bolton and I sat down recently to brainstorm on that. Here’s what we came up with:
1. Testing may die if you start using the word “testing” to mean checking. One of Michael’s contributions to the craft was to suggest a sharp distinction between testing and mere output checking. To test is to question the product so as the evaluate it. Testing is an open-ended investigation that cannot be automated. To check, however, is to gather specific information and analyze it in a manner that could, in principle, be automated. In the parlance of philosophers, checking is a mimeomorphic activity; testing is polimorphic. Some people, mainly programmers who don’t study testing much, are strongly attached to automation. In pursuing their vision of applying tools to testing, they inadvertently dumb testing down. They do with tools what tools can do. They run many checks. Testing for them becomes little more than a command-line switch on the compiler (“-t for test”, or -q for “put the quality in”). And such checks are capable of finding bugs, just not nearly the breadth and depth and variety of bugs that a skilled human can, especially if that human also uses tools in a supporting role.
Mistaking testing for checking can kill testing, in a sense, by co-opting testing practice. Testing, as Michael and I see it, would still exist, of course. But it would be relegated to rhetorical shadows.
What I mean by that is that few people would systematically learn how to test, anymore, until we came up with new words that referred to what the term “testing” once meant. To systematically learn a technical subject, you must be able to talk about it. If all your words about testing refer to shallow and mechanical processes, the deep and skilled stuff is not a part of your world.
2. Testing may die if the value of products becomes irrelevant. It dies when we don’t care about the quality of software or the people who need it. By the same token, if we always trusted the water we drank, or the meat we bought at the store, then water testing and food hygiene standards would be irrelevant. If we didn’t mind the occasional deadly fire, we’d happily see a show down at the ol’ Iroquois theater.
There really is a problem in our industry with the erosion of the expectation that anything will work reliably, ever. I was trapped outside my house, in the cold, recently, and found that my Android phone would not make any calls on the cell network. I rebooted the phone (that takes a few minutes). Still no joy. I connected to WiFi and tried to call that way but got a strange error about not being registered. I had made calls through WiFi before from my house, so I knew it could work. Finally I started Skype and IM’d my son (this was through WiFi , so why didn’t the phone calls work?). This thing is supposed to be a phone. I’m annoyed, but not surprised.
Google probably thinks I’m not going to give up my phone just because of a few glitches. This creates an opportunity for competitors to come in with a better product that kicks them out of the market— but hey— I worked for Apple, years ago, and when you’re inside a big company like that, you don’t really care. You think success is your birthright.
(Since I first wrote this I switched to an iPhone, which has been somewhat better.)
3. Testing may die if the quality of testing work is chronically poor. Unfortunately, the death of testing can be a self-fulfilling prophecy. People most likely to believe that testing is dead are — like the folks at Google —unlikely to devote themselves to the study of it. They simply don’t know how to test, or perhaps don’t care. It’s only a matter of time before management wonders why they have testers at all.
The antidote for that is a high standard of personal excellence. This is what the Context-Driven testing community stands for. We are doing our best to win over the rest of the testing world by being good role models.
4. Testing may die if all the users in the world were early adopter technocrats. Let’s pretend that all the people in the world who use computers or rely on them in some way are highly technical and tolerant of problems in the products they use. Then the need for testing would dramatically fall. Sure, they want great quality, but if they don’t get it, they understand. For minor glitches, they will have the patience to find a work around.
That may be more true for the perpetual beta products that Google famously offers (and then cancels), but everyone on Earth depends on computers in some way, even if they’ve never seen one. And a tiny minority of those people will experimentally download a tool like Google Earth, as I have, and then spend an hour re-configuring it so that it will actually run.
5. Testing may die by suffocation. If testers are forced to channel all their ideas through a limiting set of artifacts or tools, their productivity may collapse. I’m talking about elaborate test plan templates, test script templates and test management tools, Cucumber “executable specifications” or other automation tools that require the tester to express himself only in stilted and limited ways.
That will kill testing because it turns testers into tool jockeys, whose standard of success is the weight of paper or volume of data or lines of code — none of which has much to do with testing. Tools can be marvelously helpful in moderation, but the excellent tester will resist obsessions with tools, documents, or anything that systematically impedes the variety and profundity of his work.
6. Testing may die if technology stops changing. Testing is questioning the product. There isn’t much call to question a product that stays the same, especially if it operates in an environment and for a user base that also doesn’t change. The ambition to innovate is what invigorates the need for testers. Take away that ambition and we all will have to get jobs in comic book stores.
7. Testing may die by starvation. When companies reward people who take unknown risks, but not people who discover what those risks actually are, testing is not being nourished. If the craft becomes uninviting to smart, talented, motivated people because you’ve turned it into a boring, uninteresting activity: that also will starve testing. The only people left would be the ones who are too frightened or lazy to leave. The reputation of testing would become steadily worse.
Michael and I teach Rapid Software Testing, which is like a martial art of testing. It’s exciting. We are trying to show people that their jobs don’t have to suck. We feed the testers.
As with any societal transition wherein “death” becomes a mainstream analogy, there will come a large number. I don’t disagree with your thoughts, but I am a manual tester who has already been made irrelevant by automation. I am part of that big number that changes like this come with. Keep writing, but know that some of us have already been sacrificed and gave up.
[James’ Reply: The testing field is certainly ill. Not dead, but weak and sickly. Tip: don’t call yourself a manual tester. There is no such thing as a manual tester. You weren’t made irrelevant by automation unless you literally only pushed buttons and did simple output checking for a living. But if that were true, then your irrelevancy was not because of using your hands to push buttons, but rather because you were not using analytical skill. A tester’s service is in risk analysis and experiment design. Automation doesn’t do that.
But I agree that there are a whole lot of confused companies out that that have confused output checking with testing, and now that they have “automated” they have poor testing.]
I must agree with James.
It’s definitely a plus if you can do programming because you have both more “authority” in eyes of those you have to convince about the proper strategy and understanding of the pitfalls (if you have experience with automation). But it’s not by far all that is needed.
While I was hired primarily for automation, in the end the most value I provided towards the quality investigation of the product had nothing to do with automation of checks and while automation of testing is always the popular meme, at least people tend to understand more the actual tester expertise and mindset. A good tester is also advisor to the team, questioning more than just the compiled code.
I particularly relate to number 3, as in a previous role the company decided that it didn’t need testers; they had already implemented a policy of only testing apps that were developed in-house (which meant that the company bought proprietary apps that didn’t work, and then they wondered why no-one used them). This also meant that no-one ever tested new website pages.
Then they decided to eliminate in-house development altogether, especially as they were moving office and had to make economies. I was made redundant in this process; the first day I wasn’t at work, they launched the new office and announced via social media “”Click HERE for a virtual tour of our fantastic new offices!”” Of course, the link didn’t work. I admit to texting “”You’d think someone would have tested this new page before it was published. Oh no, I forgot – you SACKED all your testers!”” Schadenfreude is a dish best served piping hot.
There is one thing that might help change these opinions: due diligence. The company I am with now works in the higher education sector, and we anticipate that more of our clients will award contracts based on our achieving certain standards of professionalism. It’s informed some discussion about the professional educational standards we should be looking for our testers to fulfill so we can answer questions from potential clients about the standards the company applies. Testing to a given professional standard is the sort of thing we might expect in a battery of due diligence questions.
[James’ Reply: Ah schadenfreude. I was hired to teach, last year, by a refugee from a company I had previously trained for. His old company had also decimated their previously high-morale testing community. Just before this particular fellow jumped ship, he was approached by management to try and rebuild that testing capability because apparently the “”automate everything”” strategy wasn’t paying off, after all. But by that time the project environment had been poisoned against the very idea of skilled testing, so he left and joined some of his old colleagues at another company nearby and that’s how I got the new gig.]
I once wrote an article around this same area of concern. The analysis was meant to speak about general trends and not specific companies. I’m sure there will be specific examples to counter any of my claims. This was primarily an analysis from a US perspective. I started by dividing the world up into roughly four big sectors:
– Mobile-only
– E-Commerce (including mobile space) & Technology-First Companies
– Big Stuffy Oldschool Companies, “”IT”” & Gov’t
– Hardware
It seems to me QA as a profession is mostly dead for anyone in the mobile space. Most folks are tolerant of errors in that space and most apps are simply not expensive enough to justify testing. Race to the bottom with the biggest value being taking people’s personal data and selling it. There are a few games designed to be slot machines, addicting, but they tend to be very simple with a mechanic designed to impact us psychologically. Very little testing needed.
Of course, E-Commerce has motivation to do some QA to make sure their money paths work correctly, but in most areas quality can suffer an user driven, statistical approaches “”appear”” cost effective. This is where Microsoft went when it got rid of most of their QA. This is driven by the Whittaker school of thought & Facebook style of approach. Dev-ops and data science approaches might mitigate some of these changes as QA folks may hide under the cover of these roles. On the other hand, when there is enough competition to require some level of quality and where you can’t outsource testing to your customers, I think QA remains. I’d say that over all the number of QA folks will remain about static or slightly less, in spite the field’s growth, unless or until monopolies take over whatever space this involves and can force customers into a QA role.
Then you have “”IT”” style companies, companies that build sites and systems for themselves that are not really related to their core business. Think insurance, banking, car rentals, and grocery stores. Gov’t also falls under this category. If these systems drive sufficient profits or are sufficiently important to keep running, the system will justify testing. This is in part since the customers doing the testing is often not acceptable (it impacts revenue, conversions and reputation). Internal apps that can’t be cheaply dogfooded may also get testing. Another cause is that many of these companies are behind on technology, stuck in their ways and thus are unlikely to be using some of the ‘advanced’ techniques the more technical companies use. These companies will grudgingly hire QA.
Finally, there is hardware, be it a big iron, xbox or a mouse. These sorts of items require testing because they are expensive to recall and will likely continue to hire QA for the foreseeable future.
It seems to me this sort of analysis could be then mixed in to your analysis and develop something deeper. For example, it seems unlikely that testing will die from semantic meaning changes if “”IT”” organizations will continue to want older styles of testing. It might end up being fractured though. That doesn’t seem very new to me, schools of testing showed this fracturing years ago.
Tool jockey-ism could still be a thing though, since IT companies will eventually chase trends, often without realizing there are genuine differences in how different companies work (and why they work differently). In my and Kaner’s experience (from BBST), IT companies tend to end up with mediocre testing teams, so skilled testers talking them out of these decisions are probably rare. IT companies love copying other companies and may just suffocate or starve testing.
This is a genuine concern and the fact that you only got 3 comments (thus far) suggests people are either scared to talk about it or don’t see the same things you and I appear to be seeing. Truth be told, I don’t like my own analysis, it hints at bad times ahead. I also think it means security will go down hill (hard to imagine considering how bad it is now).
In your opinion, which of these 7 possibilities or combination of possibilities do you feel has the greatest chance of causing large amounts of harm to our profession? Has it changed between when you wrote it and now?
My analysis is more macro than yours, yet in some ways I suspect you have a better macro view than I do (I have a ‘day job’, while you consult and train). I’d be interested in hearing if you agree or disagree with the model I’ve generated.
Thank you for your thought provoking piece.
JCD
[James’ Reply: Have I met you? I don’t recall, but I’m a little surprised not to know you, since you seem to have hung out with a lot of CDT people, and the writing on your blog is impressive.
I am most concerned about the impressive advances in practices designed to diffuse, deflect, and avoid blame. Companies carry out astonishingly irresponsible experiments with their technology, leading to mass discrimination, disenfranchisement, and loss of privacy and security. Yet they get away with it. Self-driving cars, for instance, are inherently irresponsible. Not because they are unsafe, but because they covertly shift power away from citizens and toward huge corporations. The less society expects companies to be accountable, the less they need to police themselves, and the less need there is for skilled testing, since society is basically consenting to be the brutalized subjects of these experiments.]
JCD, I’m totally with you on the analysis. Have worked in the first 3 categories and seen how speed over quality ruled out testers. New technologies bringing monitoring to the first place, so from the auto detection to the releasing a fix to a Prod takes minutes (not even talking about roll back speed).
In my last company have being working a lot on the prevention of bugs by different technics, like improving feature development process, educating developers on testing mindset, running exploratory testing sessions, analysing defect type patters… All that became redundant in the eyes of management/business when they though that if they can find and fix those bugs quickly they don’t need dedicated people who would look after the quality. So, they closed testing department. It still just happened. Now they are hiring new specialists who would do more monitoring and analysis of all those bugs which often now are incidents.
In a mean time I go somewhere down on your numbers, where cost of the bug is still higher, so the earlier it is found is better
You and I spoke once in 2014 at CAST; I asked you a question after your talk. Isaac Howard, who co-writes on my blog also may have met you at CAST the year before. He worked as an AST/BBST admin for years. You and I also talked at length once about testing and checking in email then over skype a few years back after I wrote this article: https://www.stickyminds.com/article/semantics-and-risk-labels-software-testing . It would be hard to summarize that conversation, but effectively it was what you called a monster bashing session. It helped me understand much of what you said above applies to your concerns about testing vs checking (e.g. Giving machines more authority & agency than they should have in a more ethical rather than just a taxonomy.). It was good times. I’ve on occasion thought of editing our discussion into something more presentable and posting it, but I’ve simply not had the time and energy for it.
Thank you for the rest of your response. It’s both enlightening and not surprising to me at the same time.
JCD
I wonder how cold it really was out there for you to not think of knocking on the door 🙂
[James’ Reply: That only works if your son is the type of kid who hears door knocking and does something about it.]
Looking forward to attending your upcoming training (as a dev/devop/configurator/troubleshooter) but being a thinker rather than a doer I probably will find that I like the idea of testing more than actually doing testing. Maybe you could demonstrate how you can just start and not have to think?
[James’ Reply: I can do that.]
Are the heuristics you use thinking tools (such as those to come up with ideas) or do you use them for the opposite — so you can just start and not have to think?
[James’ Reply: Heuristic are problem solving tools, not necessarily thinking tools. A hammer is a heuristic for inserting nails into wood (it can solve your problem, but isn’t guaranteed to solve it), but it’s not a thinking tool (I don’t recommend banging your head with one). Sounds like what you need is the Plunge in and Quit heuristic. That’s where you pick something that looks hard to do and just attempt to do it right away. It doesn’t matter what you pick. The point is to try to do something now. You can always quit if it’s not getting done.]
James,
In my own experience in IT services and Captives (offshore tech centers of big businesses) as tester – I can say that tendency for leaving bugs unresolved or found in product during testing has increased multifold. Business stakeholders that pay for developing, testing and implementing software for their business are more willing to say “if it works now to some degree – lets go live, I have an army of maintenance programmers to take care of any bugs that my come up” Speed and beating competition by being the first – has made quality (absence of bugs that diminish the value of product or service) to take a back seat.
So value of good testing has been undermined by poor quality products that are (appear to be) accepted by general public. This tendency on the part of businesses is surely killing testing. If we don’t test well – bugs will leak to production – stakeholder will say “so what” – let us make patch release – stealthily without even bringing down the website.
What is the incentive for tester to find all those bugs that matter?
[James’ Reply: A bug that matters is a bug that testers will be rewarded for finding. What you are saying is that bug may not matter. I think that depends on the kind of product.]
Great post and good comments. I’ve come to testing late via tech writing after training as a developer initially, and think there are parallels in the disciplines. Points 5 and 7 resonate. Testing needs to be inviting to others and testers cannot become tool jockeys. I’ve seen the changes in tech writing and people who got left behind. A valued tech writer now is an subject matter expert who can write from their knowledge. Adobe publishing tool skills aren’t needed with Markdown and Wikis. I think testers provide an excellent external viewpoint who can call bullshit on developers who’ve got too wrapped up in new frameworks to solve business problems simply. I’m amazed how developers keep up with all the latest and greatest. I thought it was the frontend devs who changed frameworks like they changed their shorts, but working as much with backend devs now I see that all devs are looking for what’s new to fix what’s old.. Not surprisingly keeping up with all of it takes time off other stuff. I think that testers can go “full stack testing” to give them automation, manual, and usability testing feedback for value. Much in the same way a good tech writer can handle the whole job too.
I don’t really see this trend as a problem for just the qa role. To me, here appears to be a trend towards less roles in general. The goal being to increase collaboration and have less knowledge silos. I guess for some the utopia is to make people interchangeable. In the widely familiar po-sm-team set of roles, a lot of people would find their job gone.
Btw, when people ask me to automate testing or to be an automated tester or whatever, I usually say sure I can write autotests, they will require (manual) development and maintenance time, and will mostly help the regression testing effort. To clear up the terminology a bit.