An automated regression test may tell you something has been fixed or is still broken. It won’t tell you if a new problem has been uncovered which indeed has not been covered in any test cases.

Writing automated test cases is often a fixation with managers in their quest to cut time and budget and get a quick release. But it invites leaving a myriad of holes in the testing process and unrevealed bugs which a quick look over by an experienced manual tester would quickly spot.

That was a very nice article and it makes everyone proud of being a Manual Tester including me. I have been in the QA field for around 6 years performing both Manual and Automated Testing. I completely agree with your comments, but you know there is nothing in the world which is useless.
Also from a manager’s perspective automated testing looks to be useful and cost effective in a scenario there is a lot of regression test cases to be run. For Eg: (We have a product with different modules and say there is a change in only some part of the module. however we are not sure if this is going to affect the other modules). In this particular scenario automation testing seems to be more useful for both tester and also for manager.
Since a tester gets bored by testing the same product, since there are no changes or bugs in it. Also it would be cost effective for the manager to run a automated test instead of wasting a manual resource, whom he can employ in other areas which has undergone many changes.
Your comments please…

Regards,
Vijay

[James’ Reply: I think automation is potentially a wonderful thing. But your automation simply does not replicate all that a good human does while testing. Your automation is doing something different.

BTW, if I ever find that testing is boring, I just change how I test. It isn’t testing the same product that causes boredom, but doing the same things.] 

[James’ Reply: Hybrid automation like this can work nicely. Bear in mind what you give up, however. By having test execution computer controlled, the tester will not be injecting variations into the test coverage, which will lessen the bug finding power of the tests. The tester will also not be as mentally engaged, I bet, leading to fewer new test ideas during test execution. However, there may be a valuable tradeoff gained in terms of the ease or speed of doing complex tests.] 

[James’ Reply: I think what you’re really saying is that you have a particular test in mind that requires a specific kind of operation that you don’t know how to get using only people working through the ordinary user interfaces. I accept that. Now, I’d like you to accept that the very test process you are referring to also may reveal problems that you don’t know how to detect except by humans like you looking over the test while it is in progress. While witnessing this test, you may get ideas about how to redesign it– your software certainly won’t re-design itself. 

There is no fundamental controversy in our industry about test automation, itself. Of course we can and we will use tools to test our software. The controversy I’m discussing has to do with the relationship between humans and their tools. I am warning against the uncritical belief that to automate a test is to reproduce eveything interesting and important that a skilled human tester does when he tests. Automation simply can’t do that.]

Examples: A web service that clients will access to upload information to your DB; a console application that will transfer information between DBs; or evaluating how a site performs under load. None of these can be effectively tested by manual means.

Better yet, if management is pushing automated UI testing, a dose of reality - in terms of the work and/or programming expertise involved in creating effective automation for the above - may put the discussion on firmer ground.

[James’ Reply: For each of your examples, I can think of ways to test manually. Ultimately, the human users of those functions must take an action or see some kind of result. You wisely used the word “effectively”, however. I would certainly consider using tools to test those things better than I otherwise would be able.

One thing I like to do is use tools to create an interface that allows me to manually and interactively test. Then I can fluidly develop test ideas.]

1. A general comfort level - presuming the tests are valid - in areas where many different combinations must be excercised.
2. Time to explore and mumble the keys.

Another challenge I have in working with my non-technical colleagues is that they focus on the “point, click, record, playback” type of automation, which I despise and hesitate to even call testing. I have used record/playback as a means to understand a commercial tool, but have rarely relied on it to actually exercise the system under test, mainly because I find it inflexible as far as responding to changes in the system.

Any suggestions on handling these types of “automation evangelists”?

[James’ Reply: I’m of the same mind, Ken. I define test automation as “tool-supported testing.” I think of the typical scripted regression testing atuomation as one facet of that. I usually pull out my agile test automation slides to try and expand the imagination of people who think tools should replace human minds.]

There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate - read on to see why.

# 3 All automated tests (by design, nature, definition) are scripted (any exceptions ?)

[James’ Reply: This depends on your specific definition of automation and script. If to be automated means that the action is carried out by a machine made by humans, instead of by humans themselves, and if to be scripted means to act in a way that is determined in part by a set of instructions, then some automated tests may not be scripted, because some machines do not operate using instructions (I can use a hammer to hit a keyboard, which is a machine that is operating by direct manipulation, instead of by a script). I can also use automation to *design* manual tests, in which case it would be a partially-scripted partially automated test. There are many other examples.

In general, I would say that most people who point at test automation are going to be pointing at a fairly scripted test.]

# 4 (converse of #3) All scripted tests can be automated (if tool and Oracle willing)

So aren’t we limited by tool and oracle to satisfy rule #4

Shrini

[James’ Reply: It is not true that all scripted tests can be automated, because of the limits of what machines can do. There are straightforward computability limits, such as those that would require a computer to spend 10,000 years to solve the problem. There are also scripted commands that no one could perform, under any circumstances, including the instruction “run all possible tests” or “notice all possible bugs.”

There are many possible actions that we can script to some degree, and which a human could do in an acceptable way, but which would be very hard or impossible for a computer to do. These include “check to see if something went wrong” which is almost always an implicit part of a test script, and “record any new test ideas that occur to you while performing this test.”]