While I was reading the meaning of automation, I got multiple definitions.
One of the defintions was “self-regulated so as to meet predetermined requirements ”

[James' Reply: I don't recognize that definition. It makes no sense to me. It would seem to say that humans are automation. That's crazy talk!]
If I consider this definition to be correct, then how different is automation from manual testing?
I am not a veteran to automation testing. In fact, we use QTP to automate our functional testing.
As a process, we create test cases for our manual testing. Once the test case has a PASS status, we record/validate/check our automation script.

If what I am doing is actually “automation” however different is automation (from my context) to the explaination you have provided?
What actually is automation?

[James' Reply: I think of test automation, in general, as "tool-supported software testing." However, the general idea of automation is that work which is done by a machine instead of a person.]

- my 2 cents.

[James' Reply: Actually, if you automate a test that really was executed sapiently, first, then whatever the test is after it's automated is unlikely to be the same as before. This is true for a couple of reasons. One is that NOTHING is exactly the same twice. The more interesting reason, though, is that a sapient action is never the same as a non-sapient one, even when they look the same. If I asked you to check the contents of a window on the screen, and you noticed that the window was minimized, you would instantly consider un-minimizing the window to check it. A program, however, must be explicitly told to look for that problem. "Check the contents of the window" to a skilled human, means something far more than the same thing issued as a command to a program.]

Perhaps now I’ll have another tool to convince managers that automation is only a part of the solution.

Jerrad anderson

[James' Reply: By definition a non-sapient process cannot duplicate a sapient process. The reason to automate is not that we think we can duplicate sapience, but because we believe we gain something useful by using a tool, and we believe that sufficiently compensates us for what we lose.] 

AUTOMATION got its birth from SAPIENT Process,right ? If i am a no clear-eyed person, how can i automate a test process or any other else.

[James' reply: Yes, automation is born from a sapient process. How you can automate a process depends on the process, the technology you use, and what you mean by the term "automate".]

Lets take a look into Load Testing.(i)The way i develop a script is a SAPIENT, (ii)i will make use of my mind/wisdom to automate the script( again SAPIENT) in generating the load,(iii)my script will communicate with the other scripts deployed in different systems,(iv)generate results (v) will analyze the results.

[James' Reply: It's not clear to me what you mean by "automate the script." Are you talking about replacing a process that is done by humans with one that is done solely by a machine? If so, there are going to be differences between what humans do and what your automation does. Those differences may be important.]

I used the term AUTOMATE to describe all the above said SAPIENT tasks,like a combo.In my context, AUTOMATE is a child term to SAPIENT.

Sorry if i drive you to a wrong direction.

[James' Reply: I hope what you're trying to say is that you guide the non-sapient aspects of your testing with your personal sapience. That's what I do, to. Sapient testing may involve quite a lot of tool use.]

[James' Reply: Computer software IS automation. When we test it we are testing automation. Therefore it's hardly surprising that software testers would be familiar with and comfortable with the idea of using automation. Many testers in our industry, however, do not use automation to execute tests unattended.]

This would imply that *all* processes seemed to be algorithmic, would not require a human (skilled) being.

James - refer to our discussion on process during STAR EAST - you mentioned that “cooling of the coffee” is a process and you distinguished from “method” - dropping ice cubes. Process is what actually happens in an open systems environment with physical laws in force and human interactions influencing it in big way. Process is description of something that happened …

I am not sure if one can say all processes are algorithmic by nature. Testing surely not the one.

Most of the people when they say “process� - they confuse that to a written documentation (describing “what is *supposed� to happen�). I believe the process describes what actually happens.

>>> [Yan] …playing Chess a sapient process based on your definition? It would have to be no right? Since we already have machines that will never lose another game to another human (they still draw, yes.), the process does not “relyâ€? on skilled humans.

Yan, seem to indicate here is that since there are computers (algorithmically superb) that can defeat a skilled human chess player – Act of playing chess is a non sapient Process. There are instances where a human being defeated the computer in chess game. I disgree with his notion that playing chess is a non sapient Process
http://en.wikipedia.org/wiki/Garry_Kasparov#Chess_against_computers.

>>>>[James] … If you define chess as the problem of working within a certain set of rules to achieve a certain result, then chess was, at one time, a sapient process, and now it is not necessarily a sapient process

So a process becoming sapient is a situational thing, context based – right? A process can change from “sapient� to “non sapient� based on the context? Do you see a possibility of software Testing becoming “non sapient� in near future?

Shrini

[James' Reply: A sapient process loses its sapience when it loses its reliance on a skilled human (though I should point out that skilled human is redundant to a certain extent-- all normal humans have skills that no machine has). When a process no longer relies on sapience, it is a different process. It is literally a stupider process, but maybe that's okay. Humans can be expensive, after all.]

“If I take the first 10 characters of each line and use that as the primary key in the foo table, does the rest of the data correspnd to (blah from the database)?”

Of course, this won’t catch that wierd flicker on the screen when the answer is correct but the “help” link mysteriously disappears. Still, I want to automate those simple, brainless things, so I can spend my time on more important things.

[James' Reply: Well, not only does it not capture flickers, but the tests you just sketched out cover very little of the test space, in any sense. They may be useful, but on the other hand, they take some time to write and maintain, so you may not be saving the time you think you are saving. Test automation is another mouth to feed. To me it is not clear as a general rule that automating even tests that are non-sapient provides sufficient value. But sometimes I feel that it definitely does. I am not anti-automation. I am anti-bullshit. I use tools and partially automated testing, but I don't pretend I am replacing my sapience with a tool. What I'm doing instead is reducing the sapience of the process without replacing it with something of equivalent value, while calculating (and gambling) that the sapience I drop isn't critical to my test strategy as a whole.] 

But I think automating everything is crazy, especially at the customer-facing level. (I can see automating a _LOT_ of developer-facing tests, but that’s a different thing.)

Anyway, here’s something really interesting:

Remember how FIT was supposed to automate all acceptance tests and we wouldn’t need software testers?

The customer would “just” describe the tests in FIT?

Well, it turns out that people in the test automation world are starting to realize that FIT isn’t really very good at making sure that the software, welll … works. Instead, FIT now has other benefits:

“So the more important purposes of the business-facing tests are (a) to help explain more clearly to the programmer what the product director wants, (b) to give the programmers a tool to structure their work and know when it’s done, (c) to help the product director think through what she wants, and (d) to help people coming upon the project later to understand something about what it’s supposed to do.”

—> This is From Brian Marick’s Examplar blog (http://www.exampler.com/blog/2007/06/01/two-conference-ideas/), and it’s good stuff, but that’s not what I want to focus on. Instead, I’d like to focus on this:

“I’m not convinced that automated business-facing (acceptance) tests are, in the presence of good programmer testing, all that incredibly useful as tests—as, that is, regression tests, tests that will find bugs created when a programmer changes something here that causes a failure over there.”

—> Again, that is from Marick.

I know of multiple training vendors who suggest 100% test automation using FIT - that using FIT or FITness is the way to do it.

What’s the expression I’m looking for? … Perhaps “It looks like that ship has sailed”?

I liked the comments on testing APIs, as automated testing of an API can only proceed once you have a usable, tested API design — which cannot be done without plenty of sapient testing. Automated testing can help once you reach that point, as the automated testing can uncover implementation defects you introduce when you have to change the design of the API.

Conversely, you can automate the (non-sapient) testing of a GUI, which would be handy once you have finished the sapient testing of the GUI. Usability is only sapient-testable, but whether you lose any functionality from a change may (I repeat may) be automatically testable.

To sum up your position as I understand it (my paraphrase) “All good testing is sapient. However, you can use tools so that you concentrate on the high-level part of testing, while the tools handle the low-level, fiddly little details.”

[James' Reply: Your paraphrase is close to what I'm trying to say, but I think the division is more interesting than high level vs. details. Sapient handling of details helps us generate new ideas. My brother once clerked at a bookstore, and he had to field a lot of questions about where to find particular books. Because he was in the midst of that question stream, he learned that many people come into book stores looking for books that Oprah Winfrey mentioned on her talk show. From this he conceived an idea to establish a special table just for Oprah books. This was before bookstores generally did that sort of thing. I think if the customer questions were taken by an automated system, it would have been far less likely for anyone to have conceived of the Oprah table idea.

Working with details may be important. All I'm saying is that some things we can automate without necessarily transforming them, while other things are transformed by automation, and still others simply aren't automatable at all. We need to take these distinctions seriously, instead of laughing them off the way most managers and programmers seem to do.]

Could we call that foolish testing? Or absurd testing? Or vacuous testing? Or maybe asinine testing?

Oh, the fun I can have with a Thesaurus. I am now thinking this may be vacuous comment.

I think I like asinine testing.

If I hand a “test” over to a machine to perform an activity on my behalf, perhaps I am not testing; I am only delegating. Much the same way a manager delegates tasks to his/her sub-ordinates. The effect is somewhat the same: a distribution of energy in an attempt to accomplish as much/more in no more time it would take to do it myself. And the task is deemed completed when some authority puts the stamp on it. An oversimplification for sure, and I’m assuming that the workers can think for themselves (hopefully).

Is it silly to suggest that at the point that one re-enters the testing theater to observe the results of all that delegation - THAT is itself the test and not a moment before? It is purposeful and observable, two distinctly human traits. They require the human element. They insist upon “sapience”.

Otherwise, the computer is doing what Yan suggested: It is just running another algorithm, program or service. Without the human there to observe, suggest, imply, define, prioritize, guess, or bang; the computer is like the tree that fell down in the woods. Perhaps automation is nothing more than condensing the time when we can re-enter so that “actual” testing can begin?

[James' Reply: I have no problem calling the thing that the tool does "testing". It's just that the tool can't fulfill the entire mission of good testing. It can't do the whole job of good testing, and it can't do certain critical parts of the job. I think of my tools, from my pencil on up, as being in a partnership with me. A sapient process relies on a person, at some point, by definition. I think there can be completely non-sapient testing, it just would be really bad.]

“Intelligence” has already been irrevocably debased. Long live sapience!

Poul Anderson coined the term “sophont”, and used it as common parlance in some of his science fiction. His writing never defined it, but I take it to mean “organism capable of exerting and exhibiting wisdom”.

Wisdom–now *there’s* a loaded word. Would that we lived in a world where testers, programmers and managers could speak sincerely about it.

Not being context-driven seems most unwise to me. Thanks to you, Cem and your context-driven cohort (of which I am a proud, humble and desultory member). Attention is the limiting factor in most human endeavors. Looking for problems in any complex system is always tough. People want “sure things”, and they’ll usually settle for reassurance as a substitute.

Sorry if this post is too full of glittering generalities to be worth posting.

[James' Reply: All words are awkward. I like "sophont"  though.]

From now on, I will try not to use the term “manual testing�, except in quotes. I will practice saying sapient testing. I believe that nearly all “manual testing� is better being called sapient.

This implies that manual=sapient, and automated=NonSapient. While I will agree that you cannot automate the sapience out of a good testing process, I hesitate to tie that idea so tightly to manual or automated. In my experience, and I suspect yours as well, I have seen some fairly non-sapient manual testing!! I also consider the load testing I do to be a sapient process, but it is almost completely automated…the sapient aspect is the post test analysis of gathered data. Not to mention, this is now a heck of a marketing challenge for me…”Announcing TestExplorer, world’s first Sapient Testing Tool!!”

[James' Reply: Good manual testing is sapient. I talk about good testing, for the most part. Bad manual testing  may or may not be sapient. Since sapience can't be automated, then if an automated process is sapient, it can't be fully automated. Does that clear it up?]

I hear what you’re saying and I like the idea, but I personally would feel more comfortable using it as an additional modifier, for instance “I think you would get more value from your tests if you designed in more sapience.” I can think of perfectly valid contexts for that statement to be applied to both automated and manual tests.

Sincerely,
David Gilbert

[James' Reply: I might actually use a sentence like that. I don't understand the problem that you're worried about.]

From my experience, I can say that computers, on their own, *can* be “taught” to do some valuable and relevant “automated checking”, but that isn’t testing. At least not to me.

[James' Reply: It can be part of testing, though. It's just not the whole equation.] 

This is not to say that I’m opposed to using automation to help me test better/faster/more/cheaper. Quite the contrary, I am a huge fan of automation. I refer to this as computer-assisted testing (as Cem does).

The point is that I *try* to use computers to increase my ability to focus on what I will now be calling sapient testing, not to replace sapience or minimize the degree of sapience necessary to do good testing.

It’s good to have something better than “brain-engaged testing”, which is what I’ve been calling it up to now. Thanks Jim, thoughtful, brilliant and spot-on as always!

– Scott

–

President & Chief Technologist, PerfTestPlus, Inc.
Executive Director, Association for Software Testing
http://www.perftestplus.com
http://www.associationforsoftwaretesting.org

“If you can see it in your mind…
you will find it in your life.”

The point that I think you are making is that, even though software testing is often computer-assisted, it is not “testing” (an investigation designed to expose quality-related information of significance to a stakeholder) without human intelligence driving it and interpreting the results.

We can certainly use computers to help us design, implement, execute and evaluate the results of many tests. But to decide which tests to design and to evaluate the significance of the results, I think we have no professionally-defensible alternative to sapience. In addition, in many cases, a human has to work in detail with the product in order to develop sufficient insight into the product and its risks to profitably use the computer to semi-automatically design and implement a long series of tests.

– Cem Kaner

[James' Reply: Thanks, Cem.]