I’ve found your point very interesting but also pretty much self-explanatory. Here are some definitions of the word “automatic” from thefreedictionary.com:
“Acting or done without volition or conscious control; involuntary”,
“resembling the unthinking functioning of a machine”, etc.
Thus, all the dis- and advantages of test automation. In my case, to confirm your point, I’ve found more of the latter not in the GUI but performance test automation.

Best regards,
Dmitriy

An automated regression test may tell you something has been fixed or is still broken. It won’t tell you if a new problem has been uncovered which indeed has not been covered in any test cases.

Writing automated test cases is often a fixation with managers in their quest to cut time and budget and get a quick release. But it invites leaving a myriad of holes in the testing process and unrevealed bugs which a quick look over by an experienced manual tester would quickly spot.

That was a very nice article and it makes everyone proud of being a Manual Tester including me. I have been in the QA field for around 6 years performing both Manual and Automated Testing. I completely agree with your comments, but you know there is nothing in the world which is useless.
Also from a manager’s perspective automated testing looks to be useful and cost effective in a scenario there is a lot of regression test cases to be run. For Eg: (We have a product with different modules and say there is a change in only some part of the module. however we are not sure if this is going to affect the other modules). In this particular scenario automation testing seems to be more useful for both tester and also for manager.
Since a tester gets bored by testing the same product, since there are no changes or bugs in it. Also it would be cost effective for the manager to run a automated test instead of wasting a manual resource, whom he can employ in other areas which has undergone many changes.
Your comments please…

Regards,
Vijay

[James' Reply: I think automation is potentially a wonderful thing. But your automation simply does not replicate all that a good human does while testing. Your automation is doing something different.

BTW, if I ever find that testing is boring, I just change how I test. It isn't testing the same product that causes boredom, but doing the same things.] 

[James' Reply: Hybrid automation like this can work nicely. Bear in mind what you give up, however. By having test execution computer controlled, the tester will not be injecting variations into the test coverage, which will lessen the bug finding power of the tests. The tester will also not be as mentally engaged, I bet, leading to fewer new test ideas during test execution. However, there may be a valuable tradeoff gained in terms of the ease or speed of doing complex tests.] 

[James' Reply: I think what you're really saying is that you have a particular test in mind that requires a specific kind of operation that you don't know how to get using only people working through the ordinary user interfaces. I accept that. Now, I'd like you to accept that the very test process you are referring to also may reveal problems that you don't know how to detect except by humans like you looking over the test while it is in progress. While witnessing this test, you may get ideas about how to redesign it-- your software certainly won't re-design itself. 

There is no fundamental controversy in our industry about test automation, itself. Of course we can and we will use tools to test our software. The controversy I'm discussing has to do with the relationship between humans and their tools. I am warning against the uncritical belief that to automate a test is to reproduce eveything interesting and important that a skilled human tester does when he tests. Automation simply can't do that.]

Examples: A web service that clients will access to upload information to your DB; a console application that will transfer information between DBs; or evaluating how a site performs under load. None of these can be effectively tested by manual means.

Better yet, if management is pushing automated UI testing, a dose of reality - in terms of the work and/or programming expertise involved in creating effective automation for the above - may put the discussion on firmer ground.

[James' Reply: For each of your examples, I can think of ways to test manually. Ultimately, the human users of those functions must take an action or see some kind of result. You wisely used the word "effectively", however. I would certainly consider using tools to test those things better than I otherwise would be able.

One thing I like to do is use tools to create an interface that allows me to manually and interactively test. Then I can fluidly develop test ideas.]

1. A general comfort level - presuming the tests are valid - in areas where many different combinations must be excercised.
2. Time to explore and mumble the keys.

Another challenge I have in working with my non-technical colleagues is that they focus on the “point, click, record, playback” type of automation, which I despise and hesitate to even call testing. I have used record/playback as a means to understand a commercial tool, but have rarely relied on it to actually exercise the system under test, mainly because I find it inflexible as far as responding to changes in the system.

Any suggestions on handling these types of “automation evangelists”?

[James' Reply: I'm of the same mind, Ken. I define test automation as "tool-supported testing." I think of the typical scripted regression testing atuomation as one facet of that. I usually pull out my agile test automation slides to try and expand the imagination of people who think tools should replace human minds.]

There’s a piece of North American folklore about John Henry, who was a manual laborer during the expansion of the railroads in our country. His job was being replaced by steam-driven heavy equipment, as the railroad industry applied technology to become more efficient. The same dynamics are happening today with manual testers. We need to make sure that manual testers avoid John Henry’s fate - read on to see why.

# 3 All automated tests (by design, nature, definition) are scripted (any exceptions ?)

[James' Reply: This depends on your specific definition of automation and script. If to be automated means that the action is carried out by a machine made by humans, instead of by humans themselves, and if to be scripted means to act in a way that is determined in part by a set of instructions, then some automated tests may not be scripted, because some machines do not operate using instructions (I can use a hammer to hit a keyboard, which is a machine that is operating by direct manipulation, instead of by a script). I can also use automation to *design* manual tests, in which case it would be a partially-scripted partially automated test. There are many other examples.

In general, I would say that most people who point at test automation are going to be pointing at a fairly scripted test.]

# 4 (converse of #3) All scripted tests can be automated (if tool and Oracle willing)

So aren’t we limited by tool and oracle to satisfy rule #4

Shrini

[James' Reply: It is not true that all scripted tests can be automated, because of the limits of what machines can do. There are straightforward computability limits, such as those that would require a computer to spend 10,000 years to solve the problem. There are also scripted commands that no one could perform, under any circumstances, including the instruction "run all possible tests" or "notice all possible bugs."

There are many possible actions that we can script to some degree, and which a human could do in an acceptable way, but which would be very hard or impossible for a computer to do. These include "check to see if something went wrong" which is almost always an implicit part of a test script, and "record any new test ideas that occur to you while performing this test."]

When people say “manual test”, what they really mean is “human eyes, ears, hands, and brains are engaged in the execution of the test.” When they say “automated test”, what they really mean is “human eyes, ears, hands, and brains are not (necessarily or usually) involved in the execution of the test.” That reframe points to the fact that “automated” tests are entirely scripted tests, where “manual” tests are unscripted or exploratory tests. Automation can be used to aid exploration, but the exploratory act requires that human engagement–until we hear something different from WIRED.

Would any of us willingly get into an airplane that had only been tested by automation, and never by human test pilots? Would it be possible to test the software on a plane without assistance from automation? For most sane people, the answer to both questions would be No. So in evaluating the value of automation, we need to think about how we’re using it to assist and extend human capabilities.

I think it’s also important to underscore James point that low cost does not necessarily mean high value. This seems to be a source of some confusion for some testers and programmers in particular.

—Michael B.

[James' Reply: Good points, Michael. Thank you. I would just point out that a manual test is not necessarily an exploratory test (not on purpose, anyway, and maybe not otherwise). However, I agree that most manual tests are at least a small bit exploratory and probably more than that. A good manual test, even if partly scripted, will make use of unscriptable human faculties, to be sure.] 

Thanks for the great article. I support Test Automation Rule #1, but with one caveat. Some organizations will have a hard time swallowing it whole!

The company I work for is firmly entrenched in Brett Pettichord’s Factory School of testing. The idea is that if we write down all of the requirements, the coders code from them, the testers test from them, and we trace everything together. Once all of the requirements are tested, we’re done! Easy as pie, right? (Wrong!) Our automation efforts, therefore, consist entirely of automating really bad manual tests. Since the community is convinced that all of these tests are necessary to ensure Quality, it is difficult to help them understand that there are more effective and more efficient ways to test a system than these rigid requirements-based scripts.

Right now I’m trying to help them quickly automate their manual scripts first, so we have some breathing room to think about other strategies. I’m also working on a model-based testing solution, and researching some more API-level automated testing approaches.

I’m afraid that helping to dig the community out of the hole of manual testing will be the easy part. The hard part will be convincing the community that the whole idea of how we do testing could be drastically improved…

[James' Reply: Good point, Chris. This calls for a new rule. Rule #0: All rules for humans shall be treated as heuristics (including this one, of course).]

Note the core theme of notion - such tests are not expected to uncover any different bug other than what is coded - Check if cancel button exists in 100 different windows? Test pass
If yes - Test fail if otherwise.

Are these tests powerful? May be not.
Are these tests good in finding bugs - yes - only those bugs (deviations from expected behavior) coded in automated tests?
Are these tests important? Yes from the perspective of someone who matters.

Is it a good way to test? May be not - some of these tests could be executed as part of unit tests - in a cheaper way.

Let me make a statement: The way you state (communicate) your manual Test will decide whether it could be automated or not. The same thing will also decide whether a test case is good or bad.
This communication (hence documentation) is key aspect of Test design right?

For example - let us consider following test case

Test step or Action : launch Add Member Wizard Screen
Expected Result: Check that Cancel button is present on the screen and is in enabled state.

It is a manual test that can be automated - But is it a good manual Test? May be not.

I would be over-simplifying and overselling the value of above test if I claim that “through this automated test I am making sure that there are no major failures in the screen”. What I am checking here is the existence and state of control on the screen - nothing more and nothing less.

Shrini

[James' Reply: There can be good tests, I think, and there can be good formal tests. There can even be formal manual tests (by some reasonable definition of formal).

I don't believe any test can ever be repeated-- if to repeat means to repeat EXACTLY and in all dimensions. Even approximate repetition can be a challenge, depending on what you're trying to repeat. So, if you write down a test for someone else to perform, it can be an interesting challenge to make sure the right thing is repeated.]