Prabhat Nayak is yet another thinking tester recently hired by the rising Indian testing powerhouse, Moolya. Speaking of the ISTQB syllabus, he writes:
One such disagreement of mine is they have put “Exploratory Testing” on purely experienced based testing. James, correct me if I have got ET wrong (and I am always ready to be corrected if I have misunderstood something), a novice tester who has got great cognizance and sapience and can explore things better, can think of different ways the product may fail to perform per requirement can always do a great job in ET than a 5 years experienced tester who has only learned to execute a set of test cases. That is probably one of the beauties of ET. There is of course, always an advantage of having some experience but that alone doesn’t suffice ET to be put under experienced based testing.
You are quite correct Prabhat. Thank you for pointing this out.
The shadowy cabal known as the ISTQB insulates itself from debate and criticism. They make their decisions in secret (under NDA, if you can believe it!) and they don’t invite my opinion, nor anyone’s opinion who has made a dedicated study of exploratory testing. That alone would be a good reason to dismiss whatever they do or claim.
But this case is an especially sad example of incompetent analysis. Let me break it down:
What does “experience-based” mean?
Usually when people in the technical world speak of something as “x-based” they generally mean that it is “organized according to a model of x” or perhaps “dominated by a consideration of x.” The “x”, whatever it is, plays a special role in the method compared to its role in some other “normal” or “typical” method.
What is a normal or typical method of software testing? I’m not aware the the ISTQB explicitly takes a position on that. But by calling ET an experience-based technique, they imply that no other test technique involves the application of experience to a comparable degree. If they have intended that implication– that would be a claim both remarkable and absurd. Why should any test technique not benefit from experience? Do they think that a novice tester and an experienced tester would choose the exact same tests when practicing other test techniques? Do they think there is no value to experience except when using ET? What research have they done to substantiate this opinion? I bet none.
If they have not intended this implication, then by calling ET experience-based it seems to me they are merely making impressive sounds for the sake of it. They might as well have called ET “breathing-based” on the theory that testers will have to breathe while testing, too.
Ah, but maybe there is another interpretation. They may have called ET “experienced-based” not to imply that ET is any more experience-based than other techniques, but rather as a warning that expresses their belief that the ONLY way ET can be valuable is through the personal heroism and mastery of the individual tester. In other words, what they meant to say was that ET is “personal excellence-based” testing, rather than testing whose value derives from an explicit algorithm that is objective and independent of the tester himself.
I suspect that what’s really going on, here: They think the other techniques are concrete and scientific, whereas ET is somehow mystical and perhaps based on the same sort of dodgy magic that you find in Narnia or MiddleEarth. They say “experience-based” to refer to a dark and scary forest that some enter but none ever return therefrom… They say “experienced-based” because they have no understanding of any other basis that ET can possibly have!
Why would it be difficult for Factory School testing thinkers (of which ISTQB is a product) to understand the basis of ET?
It’s difficult for them because Factory School people, by the force of their creed, seek to minimize the role of humanness in any technical activity. They are radical mechanizers. They are looking for algorithms instead of heuristics. They want to focus on artifacts, not thoughts or feelings or activities. They need to deny the role and value of tacit knowledge and skill. Their theory of learning was state of the art in the 18th century: memorization and mimicry. Then, when they encounter ET, they look for something to memorize or mimic, and find nothing.
Those of us who study ET, when we try to share it, talk a lot about cognitive science, epistemology, and modern learning theory. We talk about the importance of practice. This sounds to the Factory Schoolers like incomprehensible new agey incantations in High Elvish. They suspect we are being deliberately obscure just to keep our clients confused and intimidated.
This is also what makes them want to call ET a technique, rather than an approach. I have, since the late nineties, characterized exploratory testing as an approach that applies to any technique. It is a mindset and set of behaviors that occur, to some degree, in ALL testing. To say “Let’s use ET, now” is technically as incoherent as saying “Let’s use knowledge, now.” You are always using knowledge, to some degree, in any work that you do. “Knowledge” is not a technique that you sometimes deploy. However, knowledge plays more a role in some situations and less a role in others. Knowledge is not always and equally applicable, nor is it uniformly applied even when applicable.
For the Factory Schoolers to admit that ET is endemic to all testing, to some degree, would force them to admit that their ignorance of ET is largely ignorance of testing itself! They cannot allow themselves to do that. They have invested everything in the claim that they understand testing. No, we will have to wait until those very proud and desperately self-inflated personalities retire, dry up, and blow away. The salvation of our craft will come from recruiting smart young testers into a better way of thinking about things like ET. The brain drain will eventually cause the Factory School culture to sink into the sea like a very boring version of Altantis.
Bottom Line: Most testing benefits from experience, but no special experience is necessary to do ET
Exploratory testing is not a technique, so it doesn’t need to be categorized alongside techniques. However, a more appropriate way to characterize ET, if you want to charactize it in some way, is to call it self-managed and self-structured (as opposed to externally managed and externally structured). It is testing wherein the design part of the process and the execution part of the process are parallel and interactive.
You know what else is self-managed and self-structured? Learning how to walk and talk. Does anyone suggest that only “experienced people” should be allowed to do that?
Would it make sense to say that ET is experience-informed testing? Seems like it’s approachable for a novice and yet could certainly improve based on having testing experience.
Also, I do try to reserve incantations in High Elvish for my personal life rather than my professional ET reporting.
[James’ Reply: Claire, seriously, what makes you think that scripted testing is NOT experience-informed? The ONLY testing not experience-informed would be the “inexperienced testing technique”, I guess.
Do either of these statements sound wrong to you:
1. Exploratory testing mixes test design and test execution and is a skilled activity that benefits from experience.
2. Scripted testing does not mix test design and test execution and is a skilled activity that benefits from experience.
I think they are both equally true. I prefer ET for novices, however, because it is more mentally engaging and requires less discipline and dedication to do well.]
Basically, they’re Taylorists. And they might even acknowledge the fact if you laid on the syrup and were gentle enough that they didn’t even see admitting that as a trap.
“[T]hey look [for] something to memorize or mimic, and find nothing.”
They’re at the wrong logical level — they want an algorithm or two and not heuristics — note that memorizing heuristics might have some payoff! — and apparently see even that “logical level” observation to be incomprehensible gibberish. Or worse, simply not something they can sell to Pointy-Haired Bosses.
Hi James, all
Thanks for the blog entry! My comment is not directly addressed to James Bach, but to the readers of the blog, thus “you” in my text refers to the reader.
ISTQB might be referring ET as “experience based” also for example for these 3 reasons:
1) In this way, they give a clear signal everyone can do testing, BUT only very experienced people can write test cases (that other people will execute). This sounds great for companies in the more expensive areas. They hire a senior to write test cases and a bunch of cheap sheep to “do the work”. EXACTLY like many companies do with developers. (I’m deliberately trying to avoid going in to “why programmers are mostly useless” discussion here. If someone is interested, ping an e-mail to my direction.) I see this as a way for ISTQB making money for themselves as they “certify” those “cheap sheeps” to be able to follow a test script made by that senior.
2) They aren’t good thinkers and believe the E comes from “experience”. Yes, this is a joke, but not only a joke. I thought they are not good thinkers in the past, but I am becoming more and more convinced they are great thinkers. Just that their goal is ONLY to make money – by any means necessary.
3) Otherwise it would make them lose money. The reader might want to ask himself where would ISTQB be needed, how would they make money, if ET would be better understood. Their tactic is well-known from wars. When a country is in a war, they need to dehumanize the opponent so their soldiers will not have problems attacking (=killing) other people. (Killing other people, can you believe it, we can do that!)
There was a whole lot of discussion of “test is dead” this year. My claim is that “ISTQB kills people”. I don’t even need to use that as a cheap marketing trick adopted from Nietzsche. I don’t need to wear a black robe to say it. The only thing needed is to look news and see when poor software testing causes the loss of lives. (I am not claiming all that is directly because of ISTQB, but I am not too far from the claim either.)
Prabhat wrote “a novice tester who has got great cognizance and sapience and can explore things better, can think of different ways the product may fail to perform per requirement can always do a great job in ET than a 5 years experienced tester who has only learned to execute a set of test cases”.
Everyone can do better than the latter person. Everyone. Even himself. He is only doing what a dev/coder should have added already in the code. (Not all requirements can be expressed easily in unit tests, but I am sure my point is still understood.)
How often have you seen a professional use scripts to be able to do his work? They might have some sort of checklists for helping not to forget important things, for example. But if you can imagine some profession where serious pre-scripting is needed, please let me know, so I can think about this again.
In my opinion, a written specific test case is mostly someones way to say “I know better how to test this than you”. Think about that the next time before writing that test case. Besides, if you can write it on paper, chances are you can write it in code.
Ok, I think I got misdirected of the actual topic already enough to stop the writing here. Thanks for reading until the end!
[James’ Reply: A long comment, but a useful one. Thanks.]
One thing that strikes me about this story: Factory school testing actually is experience-based. The people with the higher experience and up defning test plans for people with fewer experience. ISTQB it seems isn’t without a sense of irony in the end.
Hi
Good post, as always.
I would like to expand the bottom line:
ET is not a technique as it also helps you to choose which test design techniques to use in current project by constantly questioning decisions and activities.
Just as “car racing” is not a ‘driving technique’ or ‘technique for stepping on the accelerator’, but how to drive in any current race.
[James’ Reply: Good point. Thanks Kristjan.]
Yet another interpretation: could experience-based refer to the personal experience of the tester that you acquire while testing and designing tests in exploratory fashion.
Having worked with the foundation syllabus and not being able to fix any of the stuff I tried to fix, I could add a detail to this: the language used is german english, since real english was “too complex”. So using any real dictionaries trying to interpret the intention of the words may be difficult. Just remembered again that I never signed the NDA or the rights signoff paper, and could claim (at least per credits) partial ownership.
[James’ Reply: Dumbing down our craft to “German English”, whatever that is, is not the behavior of an organization truly interested in professionalism.
Experience is also acquired when you use other approaches to testing… Again, ET is not special with regard to experience.]
@Jari: “But if you can imagine some profession where serious pre-scripting is needed, please let me know, so I can think about this again.”
Moviemaking. … That is Blockbuster Movie Making.
But then there are also http://en.wikipedia.org/wiki/Dogme_95 movies etc. 😉
I want to provide some context for a comment I threw off earlier in this thread.
I said “they’re Taylorists”, expecting that James would know what I mean. But it’s likely someone reading here will just pass right by that. So:
I urge anyone who reads James Bach to get at least a little familiar with the life, work and influence of Fred W. Taylor. Google for him, read the Wikipedia entry for him, but especially try to get your hands on a copy of this book and at least skim through it:
Kanigel, Robert (1997). The One Best Way: Frederick Winslow Taylor and the Enigma of Efficiency. New York: Viking. ISBN 0-670-86402-1.
Getting acquainted with this man and others mentioned in that book, and their frame of mind, and the opposition they faced at the time, is key to understanding the origins and ground assumptions of much of the Factory School of testing.
One of the critical aspects of Taylor’s thinking was that workers were paid to work, not think. And even middle managers weren’t paid to think much. Procedures created through the diligent application of the principles of “scientific management” (a term he invented) would result in maximum efficiency, profit and overall societal wellbeing…or so he believed.
After you’ve read the book and found out just how much he wanted most workers to be more or less robots, turn your head back to the question:
/Is software production like “widget” (physical object) production?/
Can you say “yes” with a straight face?
Can you /defend/ such an assertion, or merely repeat it?
If you are a scientist or academic you tend to categorize reality and derive theories within specific fields. Now that is the opposite of a sufficient software testing approach since the tester has to take every aspect of circumstances in to account.
Hence science and certifications are corrupting the mindset of a tester! We should not lean on old ways for validation of testing!
Based on the ISTQB definition of exploratory testing, which is:
An informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.
[James’ Reply: That “definition” was mostly plagiarized from me, except they cut its head off and replaced it with the head of a chicken. ET IS NOT AN INFORMAL TEST DESIGN TECHNIQUE. It is the act of combining ANY form of test design with test execution in parallel (while learning, too).]
I would think that when they say that Exploratory Testing is experience based that they mean that the testing that is done during the exploratory session is usually determined by what has happened previously in that exploratory session, or experience. I don’t think that they meant that someone needs x amount of experience to perform exploratory testing.
[James’ Reply: It is not determined by any such thing. This definition does not say that it is. The part of it that you are referring to is the part I wrote. I was not saying that in ET our experience determines our testing. I was saying that in ET the design of a test is not fixed in advance.
Experience plays exactly the same role in ET as it plays in ST. The difference is that ET involves ongoing test design, and therefore, inasmuch as any test design benefits from experience, ET also does.]
It seems you have started a war against ISTQB. I’m tired of this.
[James’ Reply: Hi Maria. Perhaps you are new to the field, or maybe just new to my writing. The war you speak of has been going on, publicly, for more than a decade (perhaps since 1993, depending on where you start counting). I even called it a war, myself, at the STAR conference, way back around 2001. I’ve been on many panels discussing and debating this. I’ve written articles about this in IEEE Computer, among other publications.
I bet a lot of people are tired of it. I am, too. I want the ISTQB/ISEB idiots to go away. But since they won’t do that, I will have to continue as my nature demands: to stand up for truth and excellence; and to resist the forces that would demean and degenerate our craft.
You don’t have to be a combatant. You can let other people decide for you what job opportunities you will have and what standards by which you will be judged.]
If i am experienced in testing a product or module, then i may have a physical presence of script (i.e A Test case may be) or a imaginative presence of the script (i.e in my mind map). So, what should i call this… an Exploratory Scripted Testing or should i call this a Scripted Exploratory Testing ???
[James’ Reply: All testing is to some degree exploratory and to some degree scripted. So what we do is call it “exploratory” if it seems mostly exploratory; scripted if it seems mostly scripted.
However, we also use special labels for specific mixes of scripted and exploratory testing. For instance, if you have a brief instruction (a sentence or three) that controls a 90-minute block of tester time, we call that “chartered exploratory testing.” If you have a detailed procedural script, but you improvise on it, that’s “improvisational scripted testing.” If you have a set of lists or a mindmap, we call that “using a playbook.” If you stand behind someone and tell them what keys to push, that’s “backseat driving.” If you, the senior tester, work with a junior tester while HE drives, giving gentle guidance, that’s what I’ve come to call a “paired exploratory survey.”
Invent your own names for things. Exploratory testing is not a specific practice, it’s a general approach that contrasts AND mixes with its opposite: scripted testing.]
Thank you so much James, your post entry helped me to answer the question from people who is in doubt of ET though they did never do any experiment of ET.
Strictly for Testers and not for the people who test only the software or hardware or both.
There was a guy in early 20th century. Who wrote papers on Physics, He uncovered the facts by using his brain as a tool. It consumed more time for the “common people” to understand what it is?, i would call “common people” as lousy users of their tool. Parents who named their child
“Albert Einstein”. Field named him “Scientist”
Here is a guy from our era. who uncovered lots of myth about testing & designed testing . created a platform for future. Again common people come into play & say that he’s an controversial person who performing WAR. Only courage people can perform war and not the common people.
I call “James Marcus Bach” as a “Scientist” in the field of Testing.
Any intelligent fool can make things bigger and more complex… It takes a touch of genius – and a lot of courage to move in the opposite direction – Albert Einstein
Regards
R Maadireddy
Never argue with stupid people,
they will drag you down to their level
and then beat you with experience.
I feel, the post doesn’t look strong enough to condemn, criticize and explain …
Kindly tune the post where ever needed.
Regards
R Maadireddy