How Not to Standardize Testing (ISO 29119)

Many years ago I took a management class. One of the exercises we did was on achieving consensus. My group did not reach an agreement because I wouldn’t lower my standards. I wanted to discuss the matter further, but the other guys grew tired of arguing with me and declared “consensus” over my objections. This befuddled me, at first. The whole point of the exercise was to reach a common decision, and we had failed, by definition, to do that– so why declare consensus at all? It’s like getting checkmated in chess and then declaring that, well, you still won the part of the game that you cared about… the part before the checkmate.

Later I realized this is not so bizarre. What they had effectively done is ostracize me from the team. They had changed the players in the game. The remaining team did come to consensus. In the years since, I have found that changing the boundaries or membership of a community is indeed an important pillar of consensus building. I have used this tactic many times to avoid unhelpful debate. It is one reason why I say that I’m a member of the Context-Driven School of Testing. My school does not represent all schools, and the other schools do not represent mine. Therefore, we don’t need consensus with them.

Then what about ISO 29119?

The ISO organization claims to have a new standard for software testing. But ISO 29119 is not a standard for testing. It cannot be a standard for testing.

A standard for testing would have to reflect the values and practices of the world community of testers. Yet, the concerns of the Context-Driven School of thought, which has been in development for at least 15 years have been ignored and our values shredded by this so-called standard and the process used to create it. They have done this by excluding us. There are two organizations explicitly devoted to Context-Driven values (AST and ISST) and our community holds several major conferences a year. Members of our community speak at all the major practitioners conferences, and our ideas are widely cited. Some of the most famous testers in the the world, including me, are Context-Driven testers. We exist, and together with the Agilists, we are the source of nearly every new idea in testing in the last decade.

The reason they have excluded us is that they know we won’t agree to any simplistic standard based on templates or simple formulae. We know those things look pretty but they don’t help. If ISO doesn’t exclude us, they worry they will never finish. They know we will challenge their evidence, and even their ethics and basic competence. This is why I say the craft is not ready for standards. It will be years before all the recognized experts in testing can come together and agree on anything substantial.

The people running the ISO effort know exactly who we are. I personally have had multiple public debates with Stuart Reid, on stage. He cannot pretend we don’t exist. He cannot pretend we are some sort of lunatic fringe. Tens of thousands of testers have watched my video lectures or bought my books. This is not a case where ISO can simply declare us to be outsiders.

The Burden of Proof

The Context-Driven community stands for excellence in testing. This is why we must reject this depraved attempt by ISO to grab power and assert control over our craft. Our craft is still an open marketplace of ideas, and it is full of strong debates. We must protect that marketplace and allow it to evolve. I want the fair chance to put my competitors out of business (or get them to change their business) with the high quality of my work. Context-Driven testing has been growing in strength and numbers over the years. Whereas this ISO effort appears to be a job protection program for people who can’t stomach debate. They can’t win the debate so they want to remake the rules.

The burden of proof is not on me or any of us to show that the standard is wrong, nor is it our job to make it right. The burden is on those who claim that the craft can be standardized to study the craft and recognize and resolve the deep differences among us. Failing that, there can be no ethical or rational basis for standardization.

This blog post puts me on record as opposing the ISO 29119 standard. Together with my colleagues, we constitute a determined and sustained and principled opposition.

14 thoughts on “How Not to Standardize Testing (ISO 29119)

  1. Thanks James. This sort of contribution is important. You are talking about exactly the problem that struck me when I looked closer into what ISO were doing. ISO has clear rules about the need to achieve consensus, which is defined as “general agreement, characterized by the absence of sustained opposition… by any important part of the concerned interests”.

    The working group attempted to resolve the need for consensus, as you say, by defining the opponents as irrelevant. The Context Driven School, in their eyes, is not an “important part of the concerned interests”.

    That stance might look plausible to people who know nothing about the testing profession, but it doesn’t bear a moment’s scrutiny from anyone who understands how testing has progressed over the last couple of decades.

    The ISO 29119 working group might claim that the need for consensus applied only to the fine detail being discussed by those who were already committed in principle to a standard in that form. That stance would assume that all reasonable, professional testers were agreed on the need for ISO 29119. This would be a fine example of a circular argument and would lack any serious credibility. How can a standard be generally applicable in a diverse profession when the only people who developed it were from a single faction? How can consensus apply only to those who were already committed to agreeing about the big important issues?

    So, what is ISO going to do? Will it allow the ISO 29119 working group to ignore ISO’s own rules? Will it continue to pretend that consensus has been achieved when that is quite obviously not the case? Or will it concede? It is a difficult problem for them, but it is a problem entirely of their own making.

  2. Hi James,

    Today I got curious about ISO/IEC/IEEE 29119 and why it seems to make some testers I know (of) very upset. I almost bought it since I am dying to know what’s in it! In my mind I was sitting with the paper while trying to make sense of each and every sentence in it. I wanted proof that it was good or bad for me as a tester!

    Here’s about the time when I ended up on this page to read your blog and stopped feeling the urge to go buy the standard. I also came to think of the time I tried to use IEEE-829 to write a test plan (ISTQB class told me to) for a customer who asked for a test plan. It turned out that they didn’t care at all about my test plan because what they wanted help with was the actual testing. (To get ideas on how help them with the actual testing I got a good help from thank you very much!).

    What really caught my interest here was what you write about “The Burden of Proof” and I don’t fully understand it. I am more curious about what you mean with “The burden of proof is not on me or any of us to show that the standard is wrong, nor is it our job to make it right.” If you don’t mind, please help me to understand how you motivate this.

    [James’ Reply: “Burden of proof” is a philosophical idea. It refers to who is required to make the case. For instance, if you claim that I stole money from you, I am not required to prove that I didn’t. The burden is on you to show that I did. And if you claim that there are chickens on Mars, I am not obliged to accept it as true unless you provide sufficient evidence. In fact if you claim that there are definitely NOT chickens on Mars, you still have a burden of proof (albeit an easy one to attain). Only if you say “I don’t know whether or not there are chickens on Mars” is there no burden of proof on you, because you are reporting an internal mental state.

    If somebody claims that they have a standard that covers the work of all software testers, they have a burden to demonstrate that their standard is acceptable to credible people in the craft. It’s not like these standardizers were elected to represent us and can commit us to standards without consultation. Nobody elected them. They have no special standing.]

    You also wrote “The burden is on those who claim that the craft can be standardized to study the craft and recognize and resolve the deep differences among us.” Why is that?

    [James’ Reply: Because that is a claim they are making about US, Annie! They can’t simply presume to know us, can they? We don’t presume that they understand us and how we work, do we?]

    Best regards,
    Annie Rydholm

  3. I feel so strongly about ISO29119! I started my career trying to follow standards and living by ISTQB rules – it was when I stepped away from this and stopped fighting against my brain which was desperate to do some thinking that I think I became a better tester. Not only a better tester but one with some actual integrity. Adding rules to the existing state of confusion is going to slow even more testers down in ‘discovering themselves’.

    [James’ Reply: Congratulations on your birth as a thinking tester.]

    I really don’t understand how you could get a group of people together in a room (or a metaphorical room) to discuss how to make things better and come up with something like this. The idea that we should use the same ‘best practices’ no matter what the organisation, no matter what the project, no matter the mix of testers (I could go on) is laughable.
    I only have one best practice that I will follow religiously – I hire intelligent testers who can think for themselves to actually assess the project they are on and do what they think is best.
    Do you remember Esperanto? It was a language that was ‘invented’ that you could use anywhere because all countries would use it as their second language to create a universal language….sounds ridiculous and never caught on? Surprising… (although I think some people still learn it just incase, sounds familiar)

    [James’ Reply: Mi parolas Esperente, iomete. La hotelo estas luksa kaj grandaj…]

    Thanks for the post, it is reassuring to see so many testers agreeing on this topic!

    Sarah (@girlstest2)

  4. @JC – They will do what they have always done and simply proceed with their stated plan as it stands. If you look at the principals involved they all have a vested interest in perpetuating certain myths about software testing and what constitutes professionalism and “best practices” within it. These myths include but are not limited to regulatory compliance, the role of V&V activities, and the promulgation of command and control structures for cost savings. Just as with the ISTQB BOK the framers of this “standard” have apparently gone out of their way to ensure the documents are so arcane as to require specialized instruction and coaching if you wish to adhere. It also just so happens many of them offer this sort of service either as a primary means of income or as a supplementary income source.

    In order to sell this idea and thereby sell their services, they must convince the people with the checkbooks they have the right of it. If they come out and say it is but one of several methods for testing software many of those checkbooks are likely to snap closed. In order to keep the pyramid growing and the cash flowing upward they MUST make the claim they are the keepers of the true knowledge and insight. Anyone else can be branded a heretic or an apostate or a hysteric or simply foolish.

    @James – Evocative as always. I think we’ve both done this and had this done to us in our professional careers. It irritates me to no end when the snake oil salesmen and saleswomen with large advertising budgets pollute my personal work and reputation. I’ve had customers adamantly insist I be [insert standard] compliant and be able to show said compliance while working. More often than not these same clients had no idea what that meant since they themselves had never read the documents. All they knew is there was some sort of international standard out there and they wanted to make sure anything they did was compliant with it since they were sort of afraid of admitting they didn’t know what it was they really wanted from me. I’ve had commissioned articles refused because they were critical of organizations who either directly advertised or whose members were major buyers of advertisements. I’m glad there is a growing organization forming to fight this pyramid scheme. I only wish I could have been there at CAST this year as I had originally planned.

    I didn’t attend because my company sponsor pulled their backing at the last minute. The stated reason was essentially that the development engineer had never heard of CAST. They suggested I attend one of the STAR conferences instead and perhaps get some training on the ISTQB while there.

    [James’ Reply: In the words of Bugs Bunny, “What a maroon!”]

  5. I was wondering what took you so long to criticize this new standard!

    [James’ Reply: As I get older I am trying to get my students to take up the job of saving the world.]

    I think it is OK to have a rule book and then we also break the rules like real testers.

    [James’ Reply: Whose rule book? My rule book?]

    You may want to clarify if you are just disagreeing with the Test Process and Documentation part of the standard or if you are against all parts of the standards.

    [James’ Reply: I’m not disagreeing with the standard… THERE IS NO STANDARD. In order for me to disagree with it, it would have to first exist. What exists is a bunch of ignorant opinions from shadowy people.]

    In my opinion some part of the standard is still good to have since it gives a base definitions for testing and few test design techniques.

    [James’ Reply: That is not an argument for it being good. You can’t say that a document should be a standard because it has definitions in it. Please have a higher bar for your craft than that! I have definitions, too. Why don’t you call my body of work a standard?]

    It will help fresh grads to get introduced to testing.

    [James’ Reply: That’s terrible! We don’t want fresh grads getting introduced to testing on the false belief that this “standard” represents our craft.]

    When the word Standard is added, it obviously brings in more attention to the fresh grads who wants to start from basics (believe me….when i say grab and read a book on testing techniques people hardly listen, but when i say go and read this ISO standard they mostly make the attempt).

    [James’ Reply: That’s why we have to discredit that junk.]

    I have also interviewed many ISTQB certified testers who are not worth employable as testers. It’s sad part that not may universities teach the testing ideas in most part of the world. So i think, its also an important responsibility for the senior testers to introduce the fresh testers with the context driven testing approach(like my seniors did). People who deserve to be good testers will start appreciating context driven testing as they mature in the art. It doesn’t matter if we are being a good cop or bad cop, what matters is if we are able to nail down the tough culprits (Defects) for which we are being paid for!

  6. James, where can be downloaded the debated ISO 29119?

    At I found only a table of contents with no details.

    Aha, found it, it requires $194 for a copy. Damn.

    [James’ Reply: This is part of what we mean by “rent-seeking.” ISO first seeks to declare an invalid standard, then they force us to pay them money to learn about the standard, whether that be to critique it or comply with it.]

  7. Gokul: “when i say grab and read a book on testing techniques people hardly listen” in this case maybe tell them to pick up a book called “The Agile Samurai”. It’s a short read and inspires thinking, rather than NOT thinking.

    “It’s sad part that not may universities teach the testing ideas in most part of the world.” I would disagree with this. EVERY university, regardless of a students declared major or course of study, teaches you how to think. And thinking is our craft.

    James, thank you so much for this blog and for your work to promote thinking in testing. My previous manager very much promoted brainless “testing”. While under his tutelage I obtained the ISTQB foundation certification and I’m quite embarrassed to even talk about it. The more I think about it the more I realize how useless it is. “Let’s all make sure we are only looking at a problem from a single viewpoint so as to avoid finding issues that may come up from other viewpoints.”

    [James’ Reply: We forgive you. Lots of testers go through this.]

  8. As I read this blog and become more informed about what this ISO 29119 “standard” is doing, it occurs to me that prescribed standards are one of the reasons I left the field of education and entered technology, specifically software testing. The human mind is still, by far, the most powerful and interesting tool that exists to interact with, explore, and test software. The nuance of the human mind provides us as thinking, context-driven testers, the unique ability to learn about our environment and make critical decisions based upon our awareness of the environment. To me, it is not unlike a continuation of the evolution of thought, observation, and decision making that has brought humanity this far.

    I will not make a diatribe here about the multiple comparisons (and failures) that I imagine happening in software testing if ISO 29119 is indeed accepted as “standard.” In ten years as an educator in two different states (in schools serving underprivileged children), I watched as an effort toward standardization of content attempted to (and in some cases completely succeeded) in destroying teacher and student creativity at the expense of something “looking pretty” for outsiders. If this happens, as I said, I anticipate software testing and software testers going in the direction of education in the United States: a pretty book, full of ideal standards, that kill creativity, innovation, and exploration (substance) for the sake of shallow agreement.

    [James’ Reply: Thank you, Jessica. Of course, I bounced out of the school world after only 11 years in it. I couldn’t fit into their standard ways.]

  9. I think it is OK to have a rule book and then we also break the rules like real testers.

    [James’ Reply: Whose rule book? My rule book?]
    [Gokul] I meant the standard as the rule book because if the organization will be evaluated for their testing standard against a check list which is pathetic! It will be fun to watch organization following all these standards and still having customer complaints(this is what i meant by breaking their rules)!

    [James’ Reply: This does not answer my question. Whose rule book? Everything “standard” is produced by someone. Either that is adopted by other people or it isn’t. The Metric system is a standard, but so is the Imperial system. We don’t follow the metric system in the USA. It’s not our standard.

    I see your point that even following those rules won’t help them, but I don’t want to let some random child create standards for me just because it’s good for their education to fail.]

    In my opinion some part of the standard is still good to have since it gives a base definitions for testing and few test design techniques.
    [James’ Reply: That is not an argument for it being good. You can’t say that a document should be a standard because it has definitions in it. Please have a higher bar for your craft than that! I have definitions, too. Why don’t you call my body of work a standard?]
    [Gokul] I like this comment! I think they are using the word standard just to reap commercial benefits for themselves.

    “It’s sad part that not may universities teach the testing ideas in most part of the world.”
    [Miller.Y]I would disagree with this. EVERY university, regardless of a students declared major or course of study, teaches you how to think. And thinking is our craft.
    [Gokul] It is not just about thinking, its also about thinking as a tester, followed by documenting and justifying your thought process (which is important for business while working in regulated environment like mine). How many times have you said this to yourself that “why this developer is not thinking about a simple scenario and end-up providing a defective software to you to test!” Does it mean the developer had not taken a course of study? I think that is what is missing in our university teaching system today.

  10. Hi James,

    So when do you think that the ISO will introduce their new standard for writing novels? There must be hundreds of big name authors who would love to lend credibility to such an exercise, and I think that we can all agree that the process of writing novels would benefit from standardisation. After all this would help new writers get started, and give them a common language when speaking amongst themselves, and dealing with agents and publishers.

    I think that we can all agree that the above paragraph sounds absolutely ridiculous, but after reading around this topic for the past couple of weeks, these seem to be the arguments that are being trotted out by those in favour of this standard.

    These arguments do not stand up to even basic scrutiny if you think of testing as a creative process.

  11. ISO 29119 looks like to be another of ISOs effort to bring in a standard for testing like all its other standards which are achieved by organizations through certifications only(most of the time). Like James said, it increases the burden of proof/evidence and the documentation efforts would be higher than than the effort spent on intelligent testing. In my last job, apart from testing,I was also an MR for the ISO 9001 QMS and found it to be taxing and really did not see any big enthusiasm from any of the PMs/TLs to follow the same other than merely for the sake of Audits and certification. The process compliance was always seen as an additional project overhead. There was always a big gap between what they actually did to produce quality work and what they were supposed to do as per defined process in which they did not find any value..

    Then I turned to the other alternative school of thought for testing and is recommending context based testing to clients who really listen. Being in a service industry, sometimes we are forced to follow what our clients believe in and mandate as they are all aware of only what they see as “International standards”.

    And , actually from my experience, not everyone of the clients really check whether you really follow the process in every phase of Development / testing for their projects. They are satisfied as long as they see you as CMMI certified/ ISO certified etc..Only we are burdened with additional efforts of documentations, Audits, Proofs, etc..

    This 29119 has to be stopped as ISO as an Organization may not be the Industry experts on testing to define the standards on their own. If this is being implemented without consensus, then it would become a classic example of why processes always fail. Whenever a process is implemented without the buy in from those who are expected to use/execute the process and when the value adds from the process is not visible to them, a process would certainly fail…

  12. What this ISO aims at is creating a monopoly where only companies which have money but cannot compete goes and purchase these certificates to entice new customers.If you can’t beat competition kill it seems to be the mantra.
    My take :
    Do software testing exist even without these so called standards? Answer is BIGGGG YESSSSS ..
    Do these standards prescribed work for all sets of organizations ,for instance I know of a company which excels in testing Gaming software,they don’t have any standards which are prescribed by this ISO body. The customers are happy with the end result of their testing and that is all that matters and not the test case or metrics or templates or over documentatioon.How on earth can anyone doucment Gaming software or rather prepare test steps or rather for flight simulation software to just name a few.
    Testing is an art and a craft honed by skilled craftsmen which doesn’t require certifcates but what it requires is passion to test which is ampy exemplified by many many great tester’s whom I have admired and on whose blogs I write or comment on.
    If following certain set of standards (so called standards) prescribed by ISO body could result in a defect free quality software then shouln’t there be a standard for developer’s too so that they give a defect free code.
    This ISO can never ever work for simple reason that no user of a software /hardware or anything worth it will read a user manual on how to use the same.
    So the ISO could well try to implement standard USER MANUAL for user to read and comply with disclaimer that aything he/she does beyond what is stated in USER MANUAL and any defect or issues raised therein would not be accepted.
    If at all the ISO body wants to do anyting for betterment of the testing community they should stop fiddling with this.

  13. The biggest issue that I have with the ISO 29119 standard is the lack of openness.

    [James’ Reply: That’s the biggest issue? Not the fact that it is completely illegitimate??]

    Surely it is in the communities best interest to have standards open and transparent?

    [James’ Reply: I agree, but first the standard has to be a standard at all, right?]

    I imagine the way a company could then make money from the standard is through the auditing process or training. However if it was an open standard, then the auditing process would be open for competition and there might be a loss of income but this doesn’t seem like ethical and professional behaviour. It’s like me saying, “nuh-nuh” *tauntingly dangles standards document just out of reach of a tester* “I have the best trade secrets and I’ll show them to you. For a fee, how do I know they’re the best? pay me to find out”

    [James’ Reply: You keep using the word standard, but then you talk as if it’s not a standard, but rather an open source methodology. I don’t have a problem with people creating open source methodologies.]

    I was having a conversation with someone at work about this issue, they were concerned that if this standard was taken seriously by business than more testing jobs would be offshore to a cheaper labour force and we would be out of work. Would that necessarily be a bad thing if there were more jobs in places like India where they seem to have larger rates of Poverty compared to Australia?

    [James’ Reply: The point is not that we don’t want them to have jobs. They can have all the jobs they want. The point is when that happens by dumbing down our craft and by consulting companies promoting fraudulent and damaging “standards.”

    I want you to be free to practice your craft and show an employer that perhaps you are a better tester than some faceless guy in a consulting company 10,000 miles away. So, let’s oppose efforts to make the craft unsafe for anyone except Factory school robot men.]

  14. I have been testing for about 11 years now and have worked both as per standards (during the initial ISTQB days, and yes I have done that certification) and as a context-driven-tester (though unknowingly). As soon as I arrived at a stage where I had to lead a team of testers and plan testing for a project/product, I lost the connection with these so called standards. I found every project had different requirements, every client had some pre-conceived notions of kind of testing they would like to get done and every test team had their own idea about the ‘best practice’. I have worked in Service based organisation throughout and therefore often my ideas about best testing approach for a specific project have been rejected because they do not conform to Organisational standards.

    Standards curb the creativity and oppose changes. People who promote these standards often oppose context-driven-testing because they hide behind them to cover their own incapability.

    I would just like to comment that I completely understand ‘your stand’ on our craft!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>