[This post is here for historical reasons. However, I no longer use the term “test automation.” I refuse to utter that phrase any more because people keep thinking it means that testing can be automated– as if testing is digging a hole in the ground or some other mindless activity. It contributes to a de-skilling and de-motivating culture that discourages testing expertise. If you insist on using the term then read on, otherwise, consider talking about tools instead of automation. We all use tools. What you call automation is simply the use of tools.]
There seems to be a lot of confusion about this.
Test automation is any use of tools to aid testing. Test automation has been around since DAY ONE of the computing industry. And never in that history has test automation been an “automatic way of doing what testers were doing before”, unless you ignore a lot of what testers actually do.
For the same reason, a space probe is not “an automatic way of doing what planetary scientists were doing before”, but rather a tool that extends the reach of what scientists can do. Test automation means extending the reach of testers.
Test automation is not at all new. What’s comparatively new is the idea of a tester. Long ago, in the late 40’s, dedicated testers were almost unknown. Programmers tested software. Throughout the sixties, papers about testing, such as the proceedings of the IFIPS conferences, almost always assumed that programmers tested the software they built. Testing was often not distinguished from debugging. As larger and more complex systems arose, the idea of dedicated software testers came into vogue. In 1972, at Chapel Hill, the first conference on testing software was held, and the proceedings of that conference show that testing was beginning to be thought of as a discipline worthy of study apart from programming.
At that conference, I think they took a wrong turn. There was much hope and enthusiasm for the future of test automation. That hope has not panned out. Not for lack of trying. More for lack of understanding.
What they didn’t understand, and what many contemporary programmers and testers also don’t get, is that good testing is an inherently a human process– not incidentally, not accidentally, but INHERENTLY. It’s highly social and psychological. The more complex software is, the more important that humans engage intellectually to identify and solve testing problems. But the Chapel Hill conference was dominated by men trained as programmers and electrical engineers, not people who professionally thought about thinking or who trained people to think.
(Who did, you ask? Jerry Weinberg. His 1965 Ph.D. thesis on problem solving is fabulous. He had written The Psychology of Computer Programming in 1970, a number of papers about testing during the sixties, including a section on testing in his 1961 book, Fundamentals of Computer Programming. He taught problem solving classes during the 60’s too, culminating in his 1974 book Introduction to General Systems Thinking. I regret that Jerry didn’t keynote at the Chapel Hill conference, but he will at the next CAST conference, in Toronto.)
The idea of a dedicated trained tester is newer than the idea of test automation, but unlike test automation, it’s an idea that hasn’t been tried on a large scale, because tester training is so terrible.
Pretending that testing is a simple technical process of making API calls doesn’t make the boogie beasts go away. They are still there, Microsoft. My wife still needs me to troubleshoot Microsoft Office, a product produced increasingly, I’m told, by programmers who work intensively on test tools because they never learned how to test. (My colleague Michael Bolton recently ran a testing class at Microsoft, though, so perhaps there is hope.)
Test automation cannot reproduce the thinking that testers do when they conceive of tests, control tests, modify tests, and observe and evaluate the product. Test automation cannot perform sapient testing. Therefore, automation of testing does NOT mean automation of the service provided by the software tester.
In summary, test automation means applying tools to testing. Test automation is ancient, non-programming testers are a newer idea, but what the industry has not yet really tried, except on a small and local scale, is systematically training the minds of testers, instead of just calling them “test engineers” or “SDETs”, giving them tools they don’t know how to use, and hoping for the best.
(BTW, I’m a programmer. I was hand-coding machine language on my Apple II before I ever heard of things called “assemblers” that automated the process. I also ran the Borland Turbo Debugger test team on the Borland C++ project, in the early 90’s. Before that I ran a test tool development team at Apple Computer. When it comes to programmer testing, GUI test automation, and non-GUI test automation, I’ve been there and done that.
It is my experience of the profound limitations of test automation that makes me a bit impatient with the way new generations of testers and programmers who seem to think no one ever thought of test tools before they came along.)