Paul - what makes you to believe that Automation can be applied only at GUI level (though you have used “minimum” in your sentence)? As per James - automation is “use of tools - other software programs to support any aspect of testing” - so GUI centric or based autoamtion is one of many ways to think about automation.

What if I automation using a commad line interface, APIs, unit tests using JUnit framework? All these are examples on Non GUI automation. What is I use a tool such as perlclip to generate some random test data and use them for data driven tests? What if I write a program to scan a long test report to identify errors? - examples of automation application in testing tasks other than execution”

>>> Since application stability only tends to come towards the end of the project, automation can’t be finalised (or even started) until you’re near the end of your testing time on a project anyway.

From the examples I quoted above — automation can happen at any stage of testing even when application is not even ready to test — only you need to broaden the definition of automation - anything that helps human to do better testing is an application of automation

Agree?

Shrini

* You don’t get much value out of having an automatic script until the GUI of the application, as a minimum, is stable. Scripts are very fragile: if just the name or placement of an element moves between builds, the process tends to fail at that point. Are you a tester or a script wizard? Do we use the tools or do they use us?

Therefore…

* Since application stability only tends to come towards the end of the project, automation can’t be finalised (or even started) until you’re near the end of your testing time on a project anyway.

Therefore…

* Time spent setting up, refining, and running the automated test scripts is hardly ever recouped from what would have been achieved from just doing it manually. Even if you get good at writing scripts, it will come towards the end of a project not the beginning so you can’t leverage the speed/efficiency savings.

(An honourable exception goes to load-testing software.)

Things that don’t follow but have been observed:

* Even the most worthy, open-source, open-standards, access-to-big-library tools still require you to learn what is at a minimum another (dialect of a) scripting language. Operationally there’s danger that working knowledge will be concentrated in one or two of your testers. As that happens, they will slowly become testing tool maintainers and not testers.

* What is the true possibility that these scripts are at all portable, editable, or reusable between your projects? Every project is different. Remember “reusable software”? Exactly.

If you can negotiate this minefield you might find /some/ test automation techniques useful. It’s not all bad. I might expand this into a blog post myself one day but in the meantime I’ll ride on James’s coat-tails.

Somehow I can find myself in the phrase: “We Need A Better Way To Test”. Only I would not make this statement in general. It is sort of a drive for me to see what I/we can do better at the customer site. For me, that better way consists more on existing methods and knowledge and how that can be adapted in the customer’s process.

Therefore I understand your response on it:

[Quote=”James”]
He says we need a better way to test. Those of us who are serious testers have actually been developing and demonstrating better ways to test for decades, as we keep up with technology. Where have you been, Steve? Get out much do ya?

He thinks automation is the answer. What a surprise that a programmer would say that. But the same thing was said in 1972 at the Chapel Hill Symposium. We’ve tried that already. Many many times we’ve tried it.[/Quote]

One of the interesting points you are mentioning is that we are developing and demonstrating better ways to test for decades. I think our profession is one of the dynamic professions in the world. And I think we keep on developing and demonstrating for more decades, as I strongly believe we want to maintain our adaptability.

In this context I disagree with the phrase: “We need a better way to test”. It should be more like: “We need another good way to test”. As our adaptability will lead to a new way of defining approaches in their context.

Is Test Automation the solution to all questions? I don’t think so. When I started testing you had a few number of test tools. They all mentioned that they are the answer on all problems in software testing. If you take a look at the number of tools now: there are too much to make a good pick. If there is market for all those tools, though they claim to be THE Solution, why are there still more tools coming to the market. To me it seems that the existing tools are not that suitable enough. As long as this is the situation, how can test automation be the answer to challenges in manual testing.

Would I use such a statement? Yes, I might only to trigger other people think in other context, as I think there is more in testing then just performing you well know method. Would I use such a statement in the context that everything else we have learned was good and now we need something different? No way. Is test automation the answer? Also a negative answer. Though can it help? Yes it can, only when wisely used and when we know and are aware about the way the testing process can be under control. Are there better ways of testing and will we chasing them? Of course as we want to maintain our adaptibility.

[James’ Reply: Thanks for the comment.]

Better for me isn’t people who regurgitate the same old material in some rote manner to convince people they’re well versed. Better is people who often I disagree with, who challenge my own thinking, who dare to say the established way of thinking is skewed or down right wrong.

The problem is most people don’t really study and learn this profession. Like most, testers are guilty of skim reading and not understanding, not questioning and it shows. It shows in conversation, blogs, forums, CVs, everywhere.

Like many I have my own blog (the only comment is from you James! At least someone’s reading and I write discussion papers. What narks me is all the time I’ve been posting on forums, sending emails with these papers attached or blogs I posted to I can count on one hand the amount of folks who’ve called me on what I’ve written, and looking back I’ve written some ill-informed rubbish.

Getting bloggers we think are better than now will mean we’re getting better thinkers than the handful we have now, which will mean we pull ourself up the professional standings and start advancing our profession instead of being led.

I have a couple of thoughts that built while I was reading this string…..

Firstly, I’m saddened that testing is still looked upon by some as the poor cousin in the software world. It strikes emotional chords with me for several reasons, but not least because I’ve put, and I’ve seen a lot of others put, a great deal of effort and life energy into changing things, building things and trying to move things forward.

I perceive that one of the things that holds us back is that some testers do the same thing year after year, regardless of if it works, could be improved or is just plain dumb.

As I read through people’s thoughts on automation I wondered why we’ve stagnated here. If we’ve been using it for years and we’ve all had so much experience then why is this still a debate? Why don’t we all know when to automate and when not to? The value that can be created, the typical problems and the way to balance it with real people skills and human endeavour. Why is anyone even attempting to give us a call to action such as in the cited post? Why hasnt an education programme or research programme or publishing programme taken us past this kind of stuff that would be illconceived in the minds of a 1st year undergraduate?

Secondly, where can I find more information on the history of testing in a coherent form? I’d be really keen to find some sources that help piece it all together. I have experiences from ten years in the field, but they are not rounded, not comprehensive and not necessarily coherent in the overall context of our industry. I’d be very interesting in finding some true historians to our field and think it could add a lot to our community. Can you point me in the right direction?

I’d like to think that we could build a knowledge community that can help us all move things along with a coherent bulk that we havent seen before. I dont yet know how to do it - but I think this article is sparking my interest in trying to find some solutions.

Stew

Hmm. Isn’t Cem Kaner a computer science professor? I guess his students would not be eligible to join your association then. But a quick web search shows me that his undergraduate degree was not in IT - so perhaps he’s ok then. But maybe not Hung Nguyen.

I am not sure if the author of the article has ever been at the bleeding edge of testing but having made my home here whilst James was still a tester in short pants, I can say we have been chewing over automation for years. Back in the early 90’s I worked on an engagement methodology from go to whoa for OO (smalltalk) that aimed to automate everything. I added a lot of the Test/QA strategy and assisted with such arcane things as a proper grammar of action, use case notation etc… The engagement process was to deliver tightly packaged solutions and was adopted by IBM as a World Wide Engagement Methodology. Unfortunately, outside of this type of engagement I find that automation is something I might consider when I want a fast way to add some data.
If I decide to start an association of “real testers” I will not have any IT grads, Anyone certified by the “Ponds Institute” of test crap, anyone who uses metaphor or paradigm in the one sentance and anyone who has not read Kaner, Falk and Nguyen. We will not discuss automation but will compare our just enough testing engagements. (just enough to get through to beer o’clock).