Against Certification
What follows is a somewhat grumpy argument against tester certification programs. I have mixed feelings about writing this, because I know a lot of otherwise friendly people who are involved in certification. I know there are a good many organizations committed to certification. I will probably lose some business because I’m going on record opposing it. What I hope, of course, is that I will gain as much business as I lose. My ideal client is someone who wants straight talk, rather than only happy talk.
Please keep this in mind, as you read: my opinion is based on experience, yes, and it is based on my feelings and priorities and reasons. But it is also formed through the dialectic process of considering what other people have to say. I invite you to respond.
Please do not support bad tester certification programs. If you are already certified, please don’t take it seriously or expect other people to take you seriously because of it. Thank you.
I know something about this. I was involved in producing and approving the original body of knowledge for the Certified Software Quality Engineer program for the American Society of Quality, in 1995. By the end of that process, I could not endorse the program.
I have reviewed the syllabi for ISEB and ISTQB as well as the SWEBOK. I have debated these syllabi with some of the people who helped create them. Attempts have been made to recruit me into two other certification attempts, both of which I turned down.
I’m a big fan of merit. I do not have a high school diploma. (For those of you into etymology, “diploma” refers to a folded piece of paper.) I left school because I could not put up with the certification mentality of academic life. It’s an empty-headed way of trying to fill heads. It’s soured by petty politics. I am in this industry because it rewards primarily merit. I think the current crop of tester certification programs threaten that, in much the same way that the Capability Maturity Model has stalled and withered discussion about real software process improvement in the government and military sectors of the software industry.
Because I keep getting asked about this, I’ve laid out my arguments, below.
“No Single Community” Argument
- Certification is a community phenomenon. Certification is simply a clarification of community membership. Nothing wrong with that, except when the certifying agency does not actually represent the community.
- I imagine, for some occupations, there are well-established and internationally recognized organizations that speak for those occupations. Not so with software testing.
- There are many communities within the testing industry. These communities have different ideas about testing; differing values and vocabularies. Some people say there is a consensus about “good practice” in testing. There isn’t. There is no to determine consensus. There has never even been a serious attempt to form such a consensus. (A few friends getting together to agree on practices hardly counts as an industry consensus.)
- Most people who do testing for a living don’t take classes or read books. They don’t go to conferences. They are not community activists. (This impression is based on the informal polls I take in my corporate onsite testing classes and by my conversations with people who create certification programs.) Yet they may well be able to test software effectively. There is little outreach to such testers by the testing activists. The experience and creativity of most testers is therefore not being harnessed in any systematic way by people making up certification programs.
- Sometimes people tell me that there is no real controversy among testing thinkers about the true basics of testing. Then I argue with them for an hour and see what happens. I am a living existence proof of controversy, since my rapid testing methodology rejects much of traditional testing folklore. How people react to me reveals how they define their community: those who dismiss my ideas are telling me I’m not a Citizen of Testing, and thus they preserve their consensus by banishing those who do not give consent. Through the liberal banishment of anyone who disagrees, consensus can be achieved on any topic whatsoever.
- Although certifying agencies can speak only for their own organizers, their ideas are too often taken seriously by people who don’t know any better. This distorts the great conversation and debate about what testing is and should be. People who are not testing afficiandos don’t know that the testing industry is fragmented. They don’t know that certification programs don’t represent consensus. Because they don’t know, they tend to assume that all the tester certification programs are pretty much the same, and that the certifying agencies are authoritative, and that people who are not certified must not know much about testing.
- An excellent certification program would have to be based on a comprehensive study (not just a survey or opinion poll) of testers in the field and in a variety of technology sectors.
- I cannot support a tester certification program unless it identifies its community, studies its community, and acknowledges the existence of other communities. That’s why I call myself a member of the Context-Driven School of testing and that’s why I give names like Factory School or Quality School to other factions in the testing world who have refused to name themselves.
“Bad Test” Argument
- A certification process is a testing process. I believe in good testing. Therefore, I look for a certification program that effectively tests a candidate tester for relevant qualities.
- I am interested in the ability of a tester to test; testing competence. The ability to remember word definitions and pat answers about oversimpified testing situations is not the same as testing competence.
- An exam that focuses on the way words are used is therefore a poor test of testing competence. Yet, there are no skill-based tester certification programs.
- Some people who agree with me that certification exams are a poor test of competence believe that it is at least a test for interest and commitment level. That’s fine, but there are also many other ways to demonstrate interest and commitment, and better ways too, since testing certification requires so little skill to acquire.
- I am aware of no tester certification program that actually guarantees or even indicates the quality of the tester. It has not been my experience that certified testers, of any stripe, perform any better in my testing classes (which include hands-on testing exercises) than non-certified testers.
- There are better tests available: a university degree in computer science, philosophy, psychology, law, math, electrical engineering or even music. Pretty much any university degree deserves far more credibility as a certification for testers than the pathetic quicky classes that prepare people for tester certification.
- Certification exams do not measure the quality of a tester. Until they do, they merely facilitate discriminatory hiring practices.
“Chilling Effect” Argument
- Tester certification programs of ANY kind (even really good ones) necessitate a narrowing of views about what constitutes testing. The certification program designer must make choices about what’s in and what’s out.
- The mere existence of a testing “Body of Knowledge” or syllabus has a chilling effect on the development of new testing ideas that enhance or transform that notion of knowledge. This is because people will just use the syllabus instead of rethinking it, on the assumption that it is bad to rethink an answer when the answer is already known. But in an immature craft like testing, the mere existence of a syllabus does not mean that the problem of defining the craft or the skills of the craft has been solved– even for the community it purports to serve. We need experience with a variety of models of the craft and we need to test those models.
- A second aspect of the chilling effect is that committee politics interferes with innovation.
- A third aspect of the chilling effect is that vested interests, such as consultants who develop courses, have want to keep the status quo. Rejecting change lowers course development costs.
- I believe that any attempt at certification, especially if it is done by a community steeped in traditional testing folklore, risks retarding our progress toward a better future as a respectable discipline.
“Folklore is a Bad Foundation” Argument
- Every syllabus I have seen is just a collection of folklore; paved cow-paths of popular testing mythology. Where is the critical thinking about this folklore? Why should we be satisfied with hearsay as our primary research method? I won’t take the space here to go into detail on specific examples, such as the boneheaded way boundary testing is taught, or the moronic principle that all tests must have a prespecified expected result, or that finding a bug late is more expensive than finding it early. There is just too much faith and not enough critical thinking in our craft.
- I believe that true excellence in testing is not about memorization or promotion of testing folklore, but rather about general systems thinking, epistemology, and the philosophy and methodologies of empirical research.
- The testing communities I have encountered, other than my own, almost without exception, express either indifference or contempt for the cognitive processes of excellent testing. The exceptions I have encountered are only among testing communities that don’t realize that they are testing communities (such as decision theorists or artificial intelligence researchers).
- I believe that even my own community has only just begun the kind of scholarship required to develop a robust idea of how to train and assess excellent testers. We are lurching awkwardly forth, but we are making progress.
- Testing folklore has some value, but it is no basis upon which to declare ourselves a mature and functional craft.
“Exploitation is Bad” Argument
- I want my actions to contribute maximally toward the better future of testing. I want to help other people contribute creatively to the craft.
- I observe that the motivations driving certification programs are mainly economic, rather than being rooted in a desire to improve the testing field. Consultants find that they can easily sell classes that are tied to certification requirements. I’ve been in conversations with such people, before they knew I was against certification, as I listened to them tell me what a gold mine certification is for them.
- Besides economics, some people who push certification are motivated by a desire for greater influence and respect than they would otherwise receive from their peers if they had to stand intellectually naked and alone and justify their thinking. I know a good many people– even people who speak at testing conferences and write books– who in my opinion have little knowledge or competence as testers.
- There are a lot of people out there ripe to be exploited, including novices who want an easy way into the craft, or managers who want an easy formula for hiring testers.
- I feel bad about these things because I have gone through a difficult and long process of working out my own ideas about testing and grounding them in the history and traditions of organized thinking. When the get-rich-quick personalities set up shop as testing experts without any grounding other than the kind regard of foolish friends, I get depressed.
- I am against certification when certification requires exploiting the ignorant.
“No accountability” argument
- You can’t sue a tester for malpractice, because no testing certification, under the law, establishes a true profession.
- What happens to the certified tester if he does a bad job? Nothing. Does he lose his certification? No. What skill must a tester demonstrate to maintain certification? None.
- Tester certification has no teeth. It has the same legal footing as World’s Greatest Boss or World’s Greatest Dad.
“Yes There is an Alternative to Bad Certification” argument
- I do support certification programs that are designed to promote personal responsibility and protect an activity from restrictive regulation. For instance, if people were dying too often from scuba diving accidents, scuba diving would eventually be banned.
- I generally support certification programs that provide reasonable protection for consumers in an inefficient market, without posing an unreasonable burden to trade and innovation. But in the case of testing, employers don’t need protection from bad testers (because they already have protection, see below), and even if they did need it, they aren’t getting it from any of the certification programs that currently exist.
- I support any organization’s right to decide who can be a member, within the law. There is a certification process already in the testing field. It’s called a job interview. This protects employers. This can be followed up with an assessment process called watching people do their jobs. This protects employers. If you don’t know how to tell if someone is doing their job, then you aren’t qualified to be supervising that person. If you are an employer who knows nothing about testing or software development, and yet you want to have testers and software developers working for you, consider hiring a project manager who does understand these things. Alternatively, you can hire an outside company to manage things. At least they are accountable. But who is accountable if a certified tester does bad testing?
- One alternative to certification is taking the free online Black Box Software Testing course. It’s a full bore testing course, with hours of video lectures and lots of other materials.
- Another alternative is to get friends to say good things about you. Become “colleague certified.” My business as an independent consultant, trainer, and expert witness is based almost entirely on my reputation. Reputation = opportunity = money.
- I could support a tester certification program only if I thought it was honest and useful both to the tester and people dealing with the tester. Once our craft matures a lot more, I suspect I will support some form of industry-wide certification. Meanwhile, I can support certification programs that are A) identified with a particular community rather with the industry at large and B) skill-based, C) have undergone some kind of field testing process, and D) celebrate self-critical practice.
- There are no certification programs for testers that meet all of these criteria– not even BCRIT (Bach Certified Rapid Tester) which is my own program. BCRIT is still in development. I am not promoting it, yet. I may never promote it. I’ll promote only when I’m prepared to support it with cogent reasoning and evidence. I call upon other would be certificationists to do the same.
June 27th, 2006 at 3:13 am
I am agree with james.
Even i have Brainbench and ISEB-ISTQB certification on testing. What james telling us is 100% practical truth.
July 9th, 2006 at 2:44 pm
I Don’t agree with James..it is totaly wrong
Me to ISTQB & Brainbench certified in QA/Testing .. I think certified persons have that quality of mark..by clearing these certifications they have proved themselves that they are Internationaly recognized…certification is today’s need.
July 10th, 2006 at 11:56 am
Hi Shrikant,
Is it the normal behavior of a certified tester to completely ignore the content of an argument when criticizing it, as you just did? Do you also ignore the specifications for the products you test?
If you think you can find a problem in something I said, then please identify the troubling passages, and present some kind of counter-argument that answers them. I’ll publish your criticism, and I’ll respond to it.
Otherwise, I will only publish your comments to provide an example of the arrogance and shallowness of the people who push these sick certification programs.
– James
July 17th, 2006 at 1:24 am
Hi James..
If u say like this..what QAI,ISTQB,ISEB,Brainbench will do???? do u think they might stop certification programms,by seeing your article…nope never….
[James' Reply: I'm talking to you, not them. It's up to you to uphold your own standards. You know, you can buy a fake degree online. That kind of thing goes on no matter what. We don't have to take it seriously, though, do we?] Â
July 22nd, 2006 at 7:49 pm
Agreeing on the one hand, disagreeing on the other…
I find myself baffled in recent years, as more and more endorsement is given to the test engineering field here in Israel ( and the rest of the Tech world I Guess ).
My experience goes 10 years back – in those Days I was testing a Computer Game along with 10 other people, needless to say – none had any experience what so ever, let alone any actual training / certificates etc. - but the passion and will was there…
In the Past 6 years I’ve been working as a Test Leader of a small group dealing with various, ongoing, ever-changing set of projects and technologies…
None of them has an actual Certification of any sort ( CSQE / ISTQB etc. ),
All had been trained either by myself or in former companies’ “on the go” / “on the fly” brush ups with projects…
The notion of certification has risen specifically in the past 2 years– as ISTQB certification became a buzzword in our regions. ( courtesy of ISTQB Israel and SiGIST )
However, I do agree that on the surface it almost completely revolves around certain endorsements ( e.g. Mercury ), forums, conventions etc, as You, James, mention.
All managed by a certification board that also in charge / in connection of an outsourcing group…
Non the less, - the Certification it self, when accompanied by a 10-12 month training of all sorts ( e.g. Rex Black’s You-Niversity STE Program ) – does “create” and converts formerly “ignorant to the field” people to all dancing all singing Testing terms personnel that, on top of the accumulated knowledge are “stamped” with a “diploma” of some sort, that is becoming more and more common.
[James' Reply: If you really think that is so, then I must tell you that you remain ignorant of the field to this day, because you have no reason to think the terminology and the practices that Rex offers are "more and more common" except in your local community, and you have no reason to think they are "good" practices and terms when compared against the available alternatives. It sounds like you are not aware of the alternatives.
You have the right to accept Rex's particular religion of testing. Personally, I think it's an obsolete, wasteful, and inneffective religion. I think my approach is better, but that's a matter for debate. The point is, the debate is still going on, and there are alternative approaches, and these alternatives are embedded in communities. So, any honest approach to certification must be community-based and must recognize the debate. Rex doesn't do this. I peridocally request him to, but since 2002 he has refused to even acknowledge my requests.]
And as a whole – a certain group of people is starting to form up – a group of people that are paying for a certain certificate, paying for knowledge and training, that could not have been formerly, and formally gained elsewhere, “knowledge and will” that renders “Computer Science” students and other alternatives for early age testers nearly obsolete, as much of the terminology / theoretical knowledge and most important – the actual ambition and will that qualifies a person to be a SW/HW Tester was given and was “Certified” as a by product of the certification process itself.
So – although Certification as an entity might not seem enough, People that do take part in a more expansive training in the field of Testing, do require some sort of certification at the end of the process.
In conclusion,
I would not rule out the actual presence of “Certification”, but to evaluate it as a complimentary ( although one that is almost a must ) to a person that was trained for, and recognized as, a certain level of Testing Engineer / Person.
As for the actual certification itself – as most if not all of the programs eventually handles the same issues, and as the field of testing is a turbulent one to begin with, the actual will of a person to acquire this type of certification is at times more than enough to compensate for the lacking Certification process to begin with.
And the end result – More and more people that are seeing their calling as Testers are forming and building up, - and if that is not a god thing – than what is…
BR
Eyal
[James' Reply: I'll tell you what would be a good thing-- an open marketplace of ideas, responsible people who tell the truth to each other and refuse to exploit the fears of the ignorant, a craft that is accessible to anyone with ideas and the will to try them.]
August 9th, 2006 at 2:41 pm
Two people on this thread have Brainbench certifications. I’ve taken this certification too (when they posted the certification for free), and I passed with flying colours; I think I was in the 96th percentile, or some such.
The certification was a test of the ability to memorize a certain nomenclature. It had nothing to do with testing skill as I see it. Here’s one example:
[example]
“If various error reporting tools have selection tabs or indexes for status items, which status item defers the problem report?”
Choices: “Open”, “Fix immediately”, “Resolved”, “Open-Future Release”, “Closed”.
[/example]
Now: depending on the organization and how it has set up its system, “Open”, “Resolved”, “Open-Future Release”, or “Closed” could be completely valid answers to this question; I’ve worked in organizations that have used each of these. (I chose “Open-Future Release”, and mirabile dictu, it was the “right” answer.)
The real issue though is “what does this have to do with testing?” To my mind, the question has something to do with clerical work, but it has nothing to do with the thinking and reasoning parts of testing. None of the other questions addressed critical testing skills such as addressing bias, adapting to circumstances, exploring an application, identifying risk, creating test ideas, and so forth, in any kind of way that would separate the testers from the bureaucrats.
I would be very unlikely to hire a tester who had taken the test and who was using his results to try to impress me. The kind of testers that I respect would be embarrassed to use Brainbench’s certification as a means of establishing their credibility in the field.
—Michael B.
September 12th, 2006 at 7:23 am
1st of all, James - thanks for replying to several portions of my comment.
I completely agree to what you’ve mentioned - “honest approach to certification must be community-based and must recognize the debate”
Other than your Approach, Rex’ and other are all valid.
The question in our day and age is the question of “relevance” and it is completely context related.
Related to the Company, the Project, the Product, The Budget, etc. etc.
In my line of work - i.e. handling multiple projects ranging from simple pilots and demos to full blown Client/Server/Network/Security Solutions - i need to decide, at times on daily basis, the approach i’ll use, and the approach my team should be using.
Ever lasting ongoing documentation of every bit and byte is a thing of the past, but completely off the wall exploratory approach, ( even the most well educated reason/logic/error based one ) is something to be aware of and use only in certain circumstances,
at some projects i have - that is all i can afford, and at other it is complementary.
Disregarding it completely would be foolish of course as it is one of the best ways to make actual progress and build proper evaluation of SUT/AUT/DUTs when required.
Certification wise - AT some universities you’ll learn for 3 to 4 years filled with concepts/methods/opinions of one side - only to witness something completely else when you are using it in practice, however - such a diploma ( e.g. BA etc. ) is still a mandatory requirement in a wide range of organizations and companies.
But as long as no one will centralize all approaches under one leading board / certification system - the bashing between the different approaches will rage on - and the so called “ignorance” will rise - on both ends.
And no, i do not accept all of the methods / terms / work flows / types / levels I’m learning these days, - but it is good to be aware and acknowledge some of these terms only to realize that others do - and a common language must be created. - if it is by learning all the terms independently - or via certifications of all sorts - is a subjective mater - but is indeed needed.
In any way - would be interesting to hear a lecture from you some day in Israel - any chance?
perhaps even endorsed by SiGIST itself ( Pro Rex Black - but pro/endorses other approaches as well - in fact - open minded to all Test/QA Related approaches and methods ).
Thanks for the blog
Eyal
November 21st, 2006 at 2:40 pm
On November 15th of 2006, I took the Brainbench Java 2 test.
When I went into the test, I was expecting something that would test my general
knowledge of the Java language and object oriented programming. The test that I took
did not do that.
There were a large number of questions on special purpose API’s that I have never
used. There were some questions on development tools that I have never used. And
there were a large number of “brain teaser” questions on code snippets which I could
have answered, if I would have had more time.
As a result, my test score was very low (2.60).
That score shows that I’m not good at guessing at API’s and tools that I’ve never used.
And it shows that I’m not good at brain teasers with a three-minute time limit. However,
it shows ABSOLUTELY NOTHING about my Java and object oriented programming skills.
In 2002, I studied a book on Java, and then I took a college course on it. I got an A in
the course. According to my Brainbench score, I should not have been an “A” student.
For over a year, I was the sole designer and developer of web software that my employer
believes is marketable. That software involves thousands of lines of Java code, a large
number of API’s, and advanced object oriented constructs. According to my Brainbench
score, I should not be able to accomplish what I have just accomplished.
When I was in college, I took secondary education courses. Later, while I was working
for a former employer, I designed two computer courses which I taught for several
years. As a trained and experienced trainer and tester, it is my opinion that the
Brainbench Java 2 test has a number of very serious flaws.
It appears that there have been no sound scientific studies regarding the Brainbench
claim that their tests predict employee success. And it appears that there are a large
number of companies that are blindly accepting these unsubstantiated claims.
It seems to me that a sound scientific study for the Java 2 test would include the
following elements: have thousands of working and successful Java programmers take
various Java 2 tests; have thousands of inexperienced people with Java knowledge take
the same tests; for individual test takers, have tests with a large number of questions
on API’s and tools that they have never used; for individual test takers, have tests with
large number of questions on API’s and tools that they have used; for the latter, follow
their careers as Java programmers for at least five years.
It appears to me that the creators of the Brainbench Java 2 test do not know what a
typical Java programmer does, and they have no understanding of the art of testing.
I wonder how many careers have been derailed as a result of flawed Brainbench tests. I
would like to see a scientific survey on that.
[James' Reply: Thanks for providing details of this pathetic Brainbench situation.
I once took a computer-administered test of Microsoft Word skills. I took the test because the recruiting agency told me that each candidate to be sent for my review as a hiring manager would first be certified on a battery of such tests. I wanted to be sure that any test taken by a candidate was reasonable.
I failed the test, of course, despite having a lot of experience with Word. The reason I failed is that that stupid testing software only understood one way to perform actions in Word, and if the subject solves the problem using a different feature of Word, or uses the feature in a different way, they are instantly graded down on that task. There was no backing up and getting another try.
So, like you, I wonder how many good people had been denied jobs because this software had flunked them. Someone should file a lawsuit over that.]
March 10th, 2007 at 10:52 am
Hey James,
I do agree with what you have to say about certifications on a personal front.
However, I was wondering…how organizations around the world ever be able to benchmark or assess if they have the right person. Not every person standing in line is skilled enough to test just because he thinks or claims he is!
If you had a financial product to test which required 50+ ’skilled testers’, as an organization would you recruit 50 guys claiming to be ’skilled testers’ or 50 who have a certification in testing.
Personally, I don’t believe that a certification actually speaks for an individuals skills as a tester but was wondering how it works from an managment perspective …
-p-
[James' Reply: Hiring managers do job interviews. Progressive ones use behavioral interviewing and skill demonstrations. What I do, these days, is hire people for one week contracts to see how they work.]Â
March 26th, 2007 at 10:07 am
Hi James,
Your arguments are certainly very valid and current. While living in the US I have never met a professional in the industry who even cared to ask about a certification. I have been testing for over 10 years now, and I have had over 100 interviews, everything from programming, to analytical, to test techniques, to plain dumb questions, but no manager, even the non-technical ones, ever asked me about a tester certification. Things may have changed, I have been out of the country for 3 years now. In Europe though, certification is a big thing. Recently our company required everyone in the test department to get certified with ISTQB. Not only did we have to go through the foundation level certification, but also the advanced level functional tester. The classes we took were not really that bad, comparable to other seminars I’ve had in the US. The instructors were really trying their best. But when it came down to real life issues, their best answer was to learn for the test, and to ignore experience, critical thinking, solid test techniques, and other skills/information we have aquired so far. The tests, as you have stated above, only certify your ability to carefuly choose among multiple choice answers. I once had a possible answer that listed “A,B, C, D is false and D, E, F is true” (what is that supposed to prove anyway, that you have read the answer??). The “material” can be memorized by anyone, middle school level and above. There is no need to understand the software development process or know anything about computers for that matter. All you have to do is memorize it all, just like they teach you, then you can pass the test in the highest percentile. Actually, having real life test experience and knowledge will often work against you, as you may pick the “wrong” answer.
I don’t have a solution for what is happening to the test community. What I see is that some “businesses” saw money making in this certification process, so they went for it. The way it works, in Europe at least, is that you take a very expensive 3-5 day course, then you take the test. There is not very much interest in feedback and there is a strong certification lobby who is set out to convince every software making company that your ISO certification (which is also a big deal in Europe) is in jeopardy unless you get yourself a certified test department. I don’t see any real value added by this certification, not to the certified tester, who has not learned anything valuable in the process or demonstrated his/her skill through the certification, not to the employer, who is only getting a worthless piece of paper, with no guarantee of skills to go with it. My manager does not believe in the certification, neither do those above him, but it makes us more sellable to the client.
I understand HR managers dilemma of getting good, skilled testers. But, at this point, I don’t see a way around a structured interview process, designed to weed out potential failures. I really liked an article from Sticky Minds that I read recently http://www.stickyminds.com/sitewide.asp?Function=WEEKLYCOLUMN&ObjectId=12122&ObjectType=ARTCOL&btntopic=artcol, by Fiona Charles. The idea is to get creative when interviewing testers, find out if they are real pros or just wannabes. I usually like to confront candidates with a real problem we are having (or one from the past) in the test department and observe how they go about finding a solution. This tells me almost all I need to know about a tester. I am a professional in this field, I still have a lot to learn, I don’t have all the answers, but I have designed/implemented/performed test processes in all test techniques. And yet I managed to fail one of the certification tests, because I was choosing my answers according to common sense tester logic. It’s no big deal if you fail the test, you can take it again, thus getting the certification you so desire. My point is, HR managers should inform themselves about these tests and certifications, that are just as easily obtained as college degrees from obscure institutions, before they make test certification an important criteria in their recruiting process.
June 7th, 2007 at 5:58 am
Hi James,
I totally agree with your argument. I took the foundation ISTQB exam and all I did was select an answer from multiple answers. Yes I passed the exam but it didn’t give me an satisfaction since it didn’t challenge my brain at all. I would like the people who have give a software testing exam to answer me this question: How would you test something that you can not test?. This I believe is something of a challenge to your brain rather than a question like:
What is a software testing?
a: finding bugs in a software
b: checking the software quality
c: to prove that software does what it is meant to do
d: all of the above
I know that nowadays companies ask for software testing certification and I hate it because I don’t believe that it truly reflects how good you are as a tester.
June 15th, 2007 at 4:41 pm
Hello James,
I am European and before the European Union, I could not cross a border before showing my passport.
Certification is now a passport for many companies. You are not allowed to enter unless you show from which “country” your are. So have passport from “countries” like Microsoft, Cisco, Project Management and also Testing.
Of course with a good resume you can also enter a company. But that takes time to read and it is easier “customs” to look for a visa in the form of certification.
So certification in itself is useless, but it makes travelling between companies easier.
Kind regards,
Brian
[James' Reply: If it is like a passport, that's only because of the misleading way that certifications have been advertised. These certification have force only among the ignorant. We can change that. When enough of us ridicule bad certification, the ignorant people will catch on too, and bad certification will pass away.]Â
July 20th, 2007 at 12:45 am
I came to know of your existance via Brian Marick’s website.
While I agree with your comments, and sharing your dislike of certifications in general, I still think there is value in certification programmes. Not “certification” itself.
I think ISQTB’s syllabus is a good one (not great, or comprehansive) but following it, has increased my knowledge of testing. Still at a sneerable level.
[James' Reply: How do you know it has increased your knowledge and not harmed your knowledge? Which parts of it make sense and which parts don't? What if you simply taught yourself testing from first principles?
I think the syllabus is an insult to out intelligence. It was written by people who apparently know very little about the history of the craft of testing, nor about the practical issues surrounding testing.]
I think it is a mistake for people or the community to take “certification” as a mark of competence. A certification should only be taken as a semi-complete guide to the field. Yes Companies should look beyond certifications.
Otherwise how does one even start learning about testing?
BTW I was a co-trainer for a Java course where particiapnts took the Java Sun certification to complete it. I came to know of the type of questions asked in the exam, while preparing to teach, and honestly lost quite a lot of respect for Sun cerfication. Also if I sat down to give the exam I would probably end up passing, but only just.
[James' Reply: I can't pass the ISTQB test, because my sense of integrity and pride no longer allows me to answer falsely on a test. The only ways to pass the ISTQB test is to pretend you believe something that isn't true, or to ignorantly believe things are true that aren't, or I suppose you could also answer randomly and hope for the best.]
July 23rd, 2007 at 12:10 am
I was speaking of the syllabus, not the exam itself, which I have no intention of taking.
The topics to be covered include:
Testing types, Test design, testing tools, which I had no clue of till I started reading up on. Ofcourse there might be many incorrect “facts” at various places on these topics.
These topics seem to be a good base. Why do you think the topics suggeted by ISTQB, is a load of rubbish?
Ofcourse, I gree this is just a starting point and not an inication of high competence.
BTW I am not a tester, so not a testing expert.
[James' Reply: It sounds like you want me to explain my reasoning, but since you are not a tester, I don't know how to explain it so that you would understand. Suffice it to say that I think you would be better off without the certification nonsense. Even completely ignorant, you would be better off-- less to unlearn.]
October 4th, 2007 at 2:34 pm
The burden is on the hands of the hiring managers. With or without certification, if the interviewers are not qualified to measure the ability of a candidate, the candidate can get away with feigning competence.
April 15th, 2008 at 4:49 am
I dont know why the ppl are fighting on the Certification Program.Mindset will differ from person to person,So who so ever believe in these programs,should go ahead with a certification.Those who do not believe…well ok ,dont give it……what’s so fun of making so much noise.
[James' Reply: It's not just a personal choice, it's a system that intimidates testers and test managers. It preys upon the fear and doubt of the public. It's morally wrong. It cheapens the craft. Fighting bullies, for me, is the "fun of making so much noise."]
January 18th, 2009 at 9:12 am
Ranting here… I just spent this weekend studying for ISEB Foundation. I am being forced to take it for my consulting company because they believe that it makes me more “marketable” to our clients. I thought it would be memorization of some basic terms, but instead I was OUTRAGED and INSULTED over what they’re teaching. Many of their definitions are wrong and misleading. They advocate outdated and impractical testing methods. They base their arguments on poor science and sometimes myth. They even contradict themselves many times over. Not to mention all the spelling and grammatical errors — if they didn’t even review their own documentation, how could anyone trust them to judge someone else’s testing ability!? (Especially when they devote an entire chapter to Reviews).
It does a disservice to testers everywhere, and I’m surprised that they’ve managed to gain a foothold in this industry. It proves the power of marketing. Style over substance. Saying that “any standard or certification is better than none” is no excuse — it’s creating a generation of poor testers. Anyone who practices what ISEB teaches will probably harm an organization rather than help it.
The software industry needs to know! I think there should be a formal opposition front to challenge this certification, to make the industry understand that ISEB is *not* and “Industry Standard.” Stop them before they do any more damage. I’ve read many blog posts (like the one above) speaking out against it, but is there a unified organization of people against it? Can we create one?
[James' Reply: I'm glad someone besides me is angry over this. The Association for Software Testing is opposed to these certifications. But maybe someone out there is more politically active and will start the Testers Against Stupid Certification organization.]
March 30th, 2009 at 3:51 am
Hi James,
I am very glad to read this. I see that there are people in this blog who hold the same views as mine. I’ve often interviewed such ‘certified’ testers who cant even spell out very basic test scenarios, some freshers easily do it. I am amazed that there are companies that entertain such crap ‘certified’ professionals for employment. I have a friend who got a copy of such a certificate from his friend who has the same name!! And can you believe it actually worked for him !!
I have vowed that I would never get “certified” if it is this sleazy.
Regards,
Mihir
May 27th, 2009 at 7:26 am
Hi James,
well - nearly 3 years after my first post / comment to your “against Certification” post have passed.
During this time I’ve come to participate in numerous projects as well as set up a department from scratch (which consisted of mainly “newcomers”: to the Testing and Programming Field ), and Interviewed dozens on the way…
ITCB / and It’s certifications is indeed becoming a stronger force to reckon with these days - as more and more companies and solutions providers offer “certified” testers ( some have the ISTQB Certificate - while some went through 450 “academic” hours with projects etc. ) - and recently Certifications for the “Advanced” level ( Test Managers etc. ) has surfaced. – It is important to differentiate the ones that invest their precious time in gaining a profession while getting a certificate of some sort on the way from those who gain it for the certification alone…
I must confess that I’ve listed myself to an upcoming advanced level Test + Purchased Rex’ Book on the subject as well…
but my reasoning is now different than in the past…
I’m taking the “why not” approach with a grain of salt - having the Certificate - today - does not ( and will not ) adversely affect my experience / credibility etc. - and at worst - I’ve “invested” ( or spent ) - Roughly 200$ on the process.
). and other ( dare I say ) World renowned ( and definitely active and open spoken ) professionals.
at best - my Company would be able to market myself and my team’s services better ( alongside the traditional Pre-Sale Presentation ). - During the past years - I came to understanding that while experience is Key - Referring to different schools of the profession greatly assist!,
While embracing several books of so called “old school” testing - I’ve also embraced your techniques - and thoroughly watched your presentations / videos ( you are indeed a funny guy
Gaining “terminology and semantics from these schools – I gained common language with nearly every single Testing and or QA Manager I had to interact with – which greatly assisted reaching a common goal. ( I admit though that we have always pushed our own methodology and belief into the process – but mainly via the language acquired by the different sources )
So - I’m taking the do everything - be aware of everything - diversify approach based on them all - with my team and in general.
The certification in question - ( Foundation / Advanced and other ) - is indeed more of a memory Quiz than Logic - and I’ve argued greatly with my “tutor” after my Foundation exam - as I indeed marked answers which do not conform with my view, or worst yet - my actual experience… – but “showing off this piece of paper – actually opened doors for some business ( mostly in heavy players… )
On the other hand - People - who truly want to enter this Diverse and Intriguing world of Software/Hardware Testing ( and/or Quality Assurance - to a certain degree ) - find themselves with a far worse choices tree than your common Programmer…
), - but doing it with intent to learn, gain experience and actually make a difference and set a milestone in a way to a career - is a whole other story - and indeed ITCB have taken the wrong choice here…
doing the Certification for the paper alone - is just like getting an ISO 9001:2008 Certificate while not actually adhering to its guidelines ( which is easily doable - and misses the whole point
Bottom line,
James – it seems that you have developed deep antagonism to the “traditional” and the “documented”… - I believe it is not healthy.
[James' Reply: You mistake me. I have no antipathy for documentation-- just for wasteful and useless documentation. I have no antipathy for tradition-- unless it's oppressive and silly tradition. Surely you have noticed that I make specific arguments about this, rather than just attacking the idea of tradition and documentation in general?]
– on the other hand you are indeed right in referring to Certification methods and “objective” tests as misuse and abuse…
But I don’t see you actively going against it… - lecturing and blogging is nice – and I assume you find yourself in a convention or two each year… ( ? ),
Not to mention maintaining/managing “Satisfice”– and pushing the Context Driven methodology is indeed appreciated.. – but somehow along the way – ITCB and their work is overshadowing these efforts ?…
[James' Reply: What do you think constitutes actively going against it? I speak, write, teach, and consult. What's left to do? Lobby the congress?
Stupidity, however upsetting, is not illegal.]
God speed
My apologies for the lengthy entry!
Eyal
September 16th, 2009 at 1:15 am
OMG!(read this as Oh My Google,now Google save us many times than God
) This is exactly what had happened to me.I even quit a job from a company which gave more emphasis towards certification.I was given a negative remarks in my appraisal because I didn’t do any certifications.Person with certification is promoted.That company is not a service based one to attract the client with these certified testers,it is a product based one.
I am totally against these certifications.Because many of them are just concepts which can be memorized and written,you don’t need to have questioning or analytical skills to answer them.These certification no way helps to solve the problem we face in our projects.
And my next company is also giving credit to Certified people by sending a appreciation mails and encouraging them.And guess what? Cat and Mouse race is on…
[James' Reply: Well, stand up and speak out, man.]
March 13th, 2010 at 3:09 pm
Hi James,
I have to agree with You in essence that none certification could ever replace experience on any field. In the other hand, You have to understand that none certification, even University diploma, never ever was intended to replace exp. In such, certification is only reference point to knowledge required by particular program, ie.: ISTQB and should be treated only this way.
[James' Reply: I think you missed my point. I'm not complaining about the limitations of the ISTQB, but about the stupidity and injustice of it. How would you like it if I hit you in the face, and then when you complained I were to say "I never claimed that hitting you in the face was a perfect solution to my problem, nor did I claim that it would be pleasant for you. The difficulty is in you, because you don't like being hit in the face with my fist."]
Difficulty is in people. They are week beings that has to believe in something. Some believes in God others in money and anothers in politicians. So the real thorn is not in certification programs, ok they could be better, but in the way like people treated them. We can ban certifications at all but do we realy have enough strong to change milions of people?
[James' Reply: Let's begin by banning bad certification programs and worry about changing people, later.]
I want to answer on Harinder Seera question: “How would you test something that you can not test?”
Answer is very simple - You have to place object in some well defined environment and observe this env. I do not know the English equivalent but in Polish we say “Metody Po?rednie”.
May 18th, 2010 at 6:38 am
Hi James,
I’ve only been a tester for a couple of months, but so far very happy to be here. The company I work for is paying for me to complete the ISTQB course. More than anything, I’m very keen to learn and get a foothold on the craft because I want to be good, nay, awesome, at my job. I’m told I have a natural talent for testing, which is nice.
Could you please go into some more detail about what particular aspects or topics of the course content itself to be wary of? Or, if you so prefer, perhaps you could point me in the direction of some better reading material (Black Box already noted).
Thanks for your time.
[James' Reply: Testing is an open-ended investigative process. It is a social process. It is thinking about and seeing things in multiple ways so as to shed light on the status of the product and the world. It is not a task of creating "test cases" using "techniques" like "boundary value analysis" or "equivalence class partitioning."
One simple example of how I would disagree with the ISTQB: a test does not need to have a simple "expected result." A test must have an oracle, but the full oracles by which we test are open-ended and complex, not simple and closed and deterministic. Furthermore, an oracle is not something you must have at the time the test is performed, necessarily. The typical ISTQB instructor will have no idea what I'm talking about.
But it's not specific doctrinal issues that I'm worried about. It's the attitude and implications of saying "the world of testers has agreed on what testing is, and we represent them." That's completely wrong. It's a fraud. But it's a fraud that seeks to deny work to people who don't go along with it.
If you took the course and then came to work for me, I'd tell you to forget anything you learned in it. It's all just confusing vocabulary words. Instead, practice encountering complex things and investigating how they work.]
June 25th, 2010 at 12:29 am
What follows is a somewhat grumpy argument against the weblog against certification. All of the arguments that are presented are against certification in general; not against test certification. Being granted a driver’s license does not imply that one behaves in a correct manner on the road…
[James' Reply: Did you even pay attention to the article? If you're going to make an argument, make one. This is not an argument.
My arguments *are* against testing certification, because the conditions I speak of apply to the testing situation that prevails in the world. There is no significant worldwide controversy about safe driving, but there IS a worldwide controversy on what good testing means.
I could argue that a driver's license does imply a certain level of skill, considering that a practical test must be passed.
I could argue that driver's licenses involve an activity that, unlike testing, involves the personal responsibility not to behave recklessly. If you want to argue that testing DOES involve that responsibility, then in that case, let me say I feel it is deeply irresponsible of you to support a bad certification system.
I could alternatively argue that driver's licenses are not necessary and could be done away with, for the reason you just suggested. But I don't mind driver's licenses. I think they are easy to get and seem to provide reasonable value. If there were controversy around that, I would revisit that opinion.
There IS controversy around testing certification. SOME of us care a great deal about the low quality and poor morals of certifying organizations. I would like you to care, too.]
June 25th, 2010 at 11:31 am
Hi James,
Thanks for replying. Yes, I did pay attention to the article. It is just that the same could be said on for example PRINCE2 certification. Certification does not make one a project manager.
[James' Reply: No, I really think you didn't read the article carefully enough. The Community section already answers you with regard to PRINCE2.]
A tester is someone with a specific mindset, and he or she can test regardless of any certificate. Certification does not make testers (on the contrary).
[James' Reply: Did anyone claim that certification makes testers? I didn't claim that. So I don't know why you are talking about that. My Bad Test section addresses this issue, I think.]
August 12th, 2010 at 8:37 am
As a trainer in these certification processes I obviously have mixed emotional feeling towards the arguments against tester cetrification.
For me it is the not the certification itself that is the problem but more the way industry have latched onto those certificates as having real meaning.
[James' Reply: Why do you think that not having "real meaning" isn't a problem? Of course it's a problem. You really think it's fine to promote nonsense as long as people don't take it seriously??]
I have also been involved with sales training and methods during my career as well as being a developer of safety critical systems. In any walk of life, we can divide people into categories. those who think and develop their approaches to work in whatever field they are in and those who need lots of rules to follow. The best in any field are those that work it out for themselves and do not follow rules. Regrettably these people are in short supply and we can rarely find a process to uncover them.
[James' Reply: I guess you don't know how to uncover them or produce them, but I do, and I'm not the only one who knows. It's a simple question of training, leadership, and experience. I'm sorry if you don't know how to test or don't know how to train good testers, but stupid certification is not the answer to that problem.]
Businesses want a simple way of deciding who is good and who is bad. Often the real good guys are excluded because they do not not follow the rules or do not understand the need for rules in the first place. They just do a great job without necessarily knowing why or how.
[James' Reply: Business want many things. That doesn't excuse the certification people when they commit fraud by claiming that their certification means things it doesn't mean and helps in ways that it doesn't help.]
When I was working on safety critical systems we knew that quality was not something that could be tested into a product but it was essential that it was built-in to start with. This means that I am not convinced that testing per se is a valuable profession as a stand alone process for making software better.
[James' Reply: My job isn't to make software better. My job is to test it well. I need the skills to do that. I'm doing my best to ridicule and discredit people who claim to be certifying testers when they themselves are poor testers or even aren't sure that testing is worth doing.]
I do agree very strongly that IT “professionals” need to be develop real standards for work that start to make software better at achieving the needs of the business. We need a professional body with the same standards as other engineering or professional bodies such as Medical Profession, Architects etc.
[James' Reply: In order to do that, we have to do the hard work of debating what those standards should be. This is not easy, as there are many competing schools of thought about testing.]
My exposure to the testing certification bodies reinforces the BLOGs assertion that they are a group of “mates” who what things to be done their way. Their ability to “eat their dog food” and apply rigorous testing methods to the syllabi, exams questions and “correct” answers is sadly lacking. I have often questioned many of the so called correct answers by saying “How does getting a wrong answer make me a bad tester” only to have the response that “it is what the syllabus says”.
[James' Reply: Exactly, that's why we must stand up against them.]
1) So is a certification process useful, I would argue? I would say most certainly yes.
[James' Reply: I would say you haven't defined what a certification process is, so the question cannot be answered.]
2) Are the current set of exams useful in training testers? Some testers become better by taking the courses but they would otherwise be worse than useless.
[James' Reply: I would say that no tester becomes better by taking those courses. I simply have to retrain them if they work with me.]
3) Should companies use them as a measure of testing skill? Never, HR departments and lazy hirers like the simplicity.
[James' Reply: Simplicity is easy: flip a coin. It's not simplicity they like, it's the appearance of legitimacy.]
4) Could we build a better exam regime? - Most certainly but it will require more open discussion and robust criticism of the current methods.
[James' Reply: It would require that the various schools of thought collapse into one. I doubt that will happen.]
The current regime could be argued to be better than nothing as a way of helping the otherwise less skilled majority to do some useful testing work.
[James' Reply: There is no "nothing." There is the default situation that prevailed prior to the certification craze, and that was a craft that was more free and told fewer lies to itself.]
It also excludes the more talented from passing through the hoops to get jobs which has to be bad for the industry as a whole
To give the “mates” credit they did get something moving. Now the more enlightened need to grab the reigns and take over the process to develop a more meaningful set of processes and certificates.
[James' Reply: No, the more enlightened need to ridicule it until it goes out of style.]
Training for HR departments and hirers is needed to stop them from using meaningless measures about a person fitness for any work.
Forgive my spelling and randomness of the arguments. Just sat and wrote it as it came to my head.
August 20th, 2010 at 12:39 am
Hi James,
One of the best blogs I have gone through.. ever. To the point and very correct indeed. Certifications are more of memory exams than problem solving/analytical based. And its unclear how these “certified” people apply their “learnings” from these certifications in real life projects.
Cheers,
Gurudath
August 24th, 2010 at 3:27 pm
Hi James,
I once participated in a conference of such “certified testers” here in Brazil. Instead of giving lectures about best practices about software testing or sharing experiences in the field, they’re true concern was to sell consulting for getting certified. They gave a false idea that to sell your software product or service, you have to be certified, have CMMI level 5, ISO and other kinds of certification. In my diary life as tester in my company, I noticed that the customers and users want only two things in their software: 1: that it works; 2: that it is fast. Everything else is useless. In my company, and thus, my work experience, I learned that the only “real” certification that your are a good professional, is to assure that your client is happy. The way you do that, is from learning from your mistake and from the mistake of others.
Cheers,
Wilson