I can see you’ve had some run-ins with ISEB.

I can see the parallels between your philosophy of testing and Bruce Lee’s philosophy of martial arts.

It’s a sadly repeated human story - knowledge gets crystallised, and passed on without critical examination. Soon ‘the system’ is more important than the people, and defended unthinkingly.

The happy ending is this : Bruce Lee transformed martial arts forever, although it takes constant vigilance to stop the descent back into orthodoxy.

[James’ Reply: Speaking of that, Antony Marcano gives a wonderful rant about Bruce Lee and what he accomplished. I have great admiration for Bruce Lee… and Antony.] 

For someone who wants to make a career in software testing in the UK I’d say the investment in buying Cem’s book, reading it then passing the ISEB exam (total cost, less than 150 GBP) is the best investment in your early career you’ll make. Not because passing the ISEB exam will teach you anything (but reading Cem’s book will) but because it will get your foot in the door somewhere. Perhaps this is a lame situation, but there you go.

Who knows, with a sympathetic manager you can get sent off to EuroSTAR and have the chance to meet testers from all over the world and if you’re lucky have a chat with James.

Let’s not pretend that ISEB qualification is a complete education but it’s a useful first step at least. The problem I guess is thinking that it’s more than that.

[James’ Reply: Hi Chris. As I have blogged in an earlier blog against certification, I don’t even accept this faint praise defense of ISEB. If someone has been ISEB trained, and I have to work with them, then my feeling is that they are worse off than if they had no training at all. I think ISEB is promoting the equivalent of 18th century medical practices. No doctor today would suggest that a “good start” to being a doctor is to read about trauma care circa 1785.

What’s more, I have found the ISEB people to be immune both to criticism, and to any ideas produced in the last 25 years about how to do testing. The ones I’ve talked to seem to consider platitudes and sarcastic dismissals to be the height of brilliant reasoning. In one case, an ISEB trainer argued that the fact that their ideas don’t change is a sign that their ideas are correct and that the fact that I am constantly tinkering with my own methodology is a sign that it’s wrong. That’s just the argument used against Galileo…] 

With regards to ISEB, I’d like to point out that I never put it on my CV. I took the course to find out why so many “qualified” people were frankly so useless. The answer is simple, the course teaches nothing I couldn’t find out in about ten minutes of talking to a good tester and in my experience has been taken by people who want to get more money out of their careers. The qualification does more bad than good to our industry. I would prefer a fresh graduate who is eager to learn than an ISEB qualified tester. I’m feeling queezy about even thinking about interviewing another ISEB candidate.

Corey points out that a web service has no “user interface”, and therefore is accessible only via a tool, and anything that is accessible only by a tool must therefore require “fully automated” testing.

Just to be clear on this (and I don’t want to make an enemy of Corey by saying so), I’d contend that a web service does have a user interface, and a human user interface at that. The interface is something that people call its “application program interface”, or API. When people call it that, there are some words missing but implicit. I’ll try clumsily to replace them here: a means by which an application program may interface with the web service in a way that some human requires. No program is worth testing–or writing–unless it provides some benefit to some human at some point. As you’ve argued, James, all human interaction with a computer is to some degree automated. But it works the other way too; whenever a program uses an API, it is doing so in the ultimate service of some human.

I frequently test my own code and library code via an interpreter (Ruby’s, for instance). That code may not have a GUI, but it for sure has a UI.

—Michael B.

People with emotional issues are confused and in need of counseling. Not people who assert all API tests should be automated. I don’t think all API tests should be automated. It’s just an opinion. Maybe we need to have a discussion about this and not name call. Unless this is a private joke between you and Corey. Is it?

Is it that people maybe get a machine talking to a program confused with automation?

Right, am really going now. Good weekend all. J

[James’ Reply: I do get sardonic when confronting bullies. Corey should know better than to use phrases like “fully automated” in relation to testing. And he should not ridicule people who are trying faithfully to reconcile the human element with the tool element unless he wants to be publicly scolded. That said, I do actually believe that he needs counseling– test methodology counseling.]

About testing in an iterative manner. It is friday and I am bit tired so I will refer you to me blog for more info. Hope this is OK with you James B. You may also like to read a paper of mine about agile performance testing, it will answer many of your questions I think. There is not much literature about v-model to iterative transitions. James (Bach) may suggest some more. If you drop me a line I will get a copy of the paper to you, I can’t post it anywhere because IEEE have the copy right.

http://jamiedobson.co.uk/?q=node/36

Olly. Automation is an interesting subject. The context really does matter. If you are going to test an armchair for strength you may want to use a machine to help you. If you want to check it for comfort a selection of user doing ‘manual’ tests can help. The real trick, that makes most people shit in their pants, is that you must use your brain and judgment to decide what to do and when. This takes skill, patience, and a smart mind. No certification in the world can give you this.

Sounds like you’ve done alright with our testing. Gonna write anything about it?

I wish you all nice weekends. Nice conversation.
Jamie

I too have read Lisa’s book and started a thread last year on a yahoo forum group that she was posting to regarding the chapter on test automation. The general consensus there I believe was that test automation was really the only way to move forward on iterative projects. I disagree.

[James’ Reply: Yeah, I disagree, too! Sounds nutty.]

We’ve been running a programme for over two years where there is very little automation, but we rely on skilled testers and session based testing. This has proved to be far more benificial. However, it is appropriate to mention that Lisa does follow her chapter on manual testing with another one that explains her view point. There are of course other issues around test automation such as they only ever run the same tests, so the subset of inputs and expected outputs is never really varied.

[James’ Reply: If you know anywhere that Lisa has discussed testing skills, promoted testing skills, or exhibited testing skills, I’d like to see that.]

On another point of note, I know that you class a court stating that you are a software testing expert as being better than a testing certification programme. I don’t disagree. However, you should bear in mind that in the UK Professor Roy Meadows founded the theory that three cot deaths meant murder and a great many people went to prison for murders that they may not have committed. This so called “expert” has now been discredited and the CPS is having to review many of the cases. Personally I’m not in favour of the ISEB, I’ve sat the exam and my grandmother could have passed it.

Olly

[James’ Reply: Well, if I gave the impression that being declared by a court to be an expert is better than ISEB, then I apologize. Neither have substance. The court was not qualified to declare me an expert, in any deep sense of the word. No one is qualified to do that, in my opinion. The people I respect most in the industry will tell you that they are not equipped for that. What we have is an industry full of self-promoters. I also am a self-promoter. I hope that the way I promote myself– which includes arming my clients against over-estimating me– is more credible in the long run, and helps me avoid complacency.] 

I am the author of the original comment and quote from ISEB.
In the UK it is a very good thing to have the ISEB qualification as it is a requirement for
most testing jobs. I know it is heavily based on V model and is based on theory not practice. I dont think I have ever worked on a project that has followed more than a few bits of it.
MY point was to get a discuusion going on how we (my company) are going to test as our organisation has just switched from v model to SOA and agile. None of us have experience of this so I am trying to feel my way.
I dont think I’m ever going to get my head around automating 100% of testing and never doing any manual testing. 99% of testing maybe:) but I find defects when I dont follow scripts manual or automated so the human element should remain.
I have recently wrote a reply to a tester on the original forum and I would be greatfull if you could post any helpfull info there so I dont need to follow loads of bloggs as well
I’m going to decline to reply to Corey as I have found his replies confrontational and very un-helpful so far.

[James’ Reply: Corey and I just finished a private conversation that did not go well. He seems to think I completely misunderstood your message and his reply, but he will not tell me what and how specifically I have misunderstood. Oh well. My assessment is anyone who says that good testing can be fully automated is deeply confused and needs counseling.] 

You wrote: “How we test such a thing is to write or employ a program that does have a human interface to manipulate the web service”. Good one mate, I’d agree with that. Remember we spoke about Agile Performance Testing last year? One of the techniques I spoke about was “automatic manual testing”. This involved creating a web application which the developers, deployers, and testers could use to load data, run scripts, check the system was integrated, etc. So, some tests were partially automated like this, but the real trick was that this system enabled quicker manual testing.

And we were testing… a web-services API. Hey presto.

For the rest, I am not sure if you are splitting hairs. Many developers are absolutely obsessed with automatic testing. I am not so keen myself, it’s a context thing, I certainly don’t bleat out like a sheep from Orwell’s Animal Farm “automated good, manual baaaaaaaaaaad”. But, you testers are much better at saying what should be automated and when. I think the agile lot have a lot to answer for here. You ever read Lisa Crispin’s book about testing on an agile project? One of the chapters was called “manual testing” and it was about one line long “don’t do it”. That pretty naïve approach has formed into a strong cultural meme now, especially amongst developers…

Good luck trying to get rid of it.

Jamie.

[James’ Reply: Yes, I’m familiar with Lisa Crispin’s book. It’s a depressing example of the problem. I’ve met Lisa and I’ve tried to talk with her about this… didn’t help.]