Exploratory Testing Dynamics Cheat Sheet
I’ve uploaded to my website a reference I’m developing to evaluate the quality of an exploratory testing process. It’s called Exploratory Testing Dynamics.
This document helps me determine if there is any important thought process conspicuously missing from a tester’s practice of exploratory testing. It also helps me explain to people how structured, complex, and fascinating the phenomenon of ET is. This matters because the most interesting parts of ET are intangible.
August 29th, 2006 at 10:43 pm
Exciting and confusing at the same time! I’m trying to understand the purpose of the cheatsheet. Are you using it to evaluate an established ET process? Or is the intent to use it for screening potential testers?
[James' Reply: It's an experimental tool that I use to inventory the elements of ET that are present in a tester's practice. I can use that information to diagnose ET pathologies in an organization, or an individual tester. I've used it so far to help me coach individual testers. It's a work in progress.]Â
One of the items I really liked was the last section where you listed general testing techniques. These seemed to simplify some complex, and arguably, difficult to classify, testing methods.
[James' Reply: The last section is a condensed version of the full test strategy model that is posted on my site. It focuses on testing dynamics, independent of exploratory dynamics.]
August 30th, 2006 at 12:26 pm
I like it, a very useful piece. Well done yet again.
As a note it’s interesting to see that you list genetic algorithms as something that tries and throws badness away and then advocate maintaining a bone yard. GA’s in my experience do not maintain such a bone yard of forgotten/tried population members, or if they do then this is often cyclical in that very poor members are removed for newer potentially better members - unless of course the population grows in size forever.
[James' Reply: Consider the human genome. Talk about a boneyard. It's brimming with anachronisms! Anyway, a genetic algorithm is an example of overproduction and abandonment. But, I can think of other examples, and one might be to write throwaway code to solve various problems, then maintain, as I do, a directory of all my old Perl scripts. I sift through my boneyard when I need a new program.]
September 2nd, 2006 at 3:08 am
Am a little surprised at the lack of some type of test results or response data in the ‘Evolving Work Product’ section. Though it is called ‘Evolving’, it seems that most of this section addresses preliminary or planning work.
For ‘Exploration Skills and Tactics’, is pairing really required in order to do exploratory testing well? I like doing it, but is it pathological if testers do not work in pairs or groups?
Also don’t quite get the purpose of the polarities section. Seems kind of self evident…doesn’t it? Could you please give a concrete example of how we might use the ideas in this section to evaluate a testing process?
[James' Reply: These are interesting questions. Thank you. One thing to keep in mind is that this document is but the tip of a large iceberg about which I teach. It really needs a big article to explain it. A lot of context is missing. I just thought it might be sufficiently intelligible to share with you.
Regarding evolving products, I'd like to know what specific test results or responses you are missing. In my view of testing, evolution of those work products continues throughout the entire project, not just in some sort of "planning" stage. In any case, evolution is fundamental to the idea of exploratory testing. So, if you didn't know that before, now you know. If someone claims to be doing exploratory testing, and there is no evolution of work products, that's a pathology. Furthermore, I find it worthwhile to focus on which specific things are evolving more than others.
In response to private criticism by Jonathan Kohl, I have just changed "pairing" to "collaboration". And yes, I think collaboration is essential to excellent exploratory testing. Let me put it this way, whatever I can do on my own, no matter what it is, I believe it is highly likely to be significantly improved by working with at least one other person (almost regardless of the skill level of that person). A pathology in individual ET is tunnel-vision. I agree that testing alone can lead to very good results, however.
The polarities section relates to alternation. Alternation is a bit of a subtle subject. I only discovered the importance of alternation recently. The key idea of alternation is that sometimes working on one thing for a while creates tension that is resolved by working on its opposite. A simple example is when pulling on a sock, I pull up on the front of the sock until it binds up, and then I pull on the back of the sock, etc. I alternate back and front. Pulling on one side creates the conditions that free up the other side. The same kind of thing happens in problem solving. So, if I asked you to evaluate the robustness of a tester's use of alternation, what would you do? There are so many types of alternation, it's overwhelming. To help myself evaluate alternation behavior, I find it useful to have a list of reminders about the different things that might be alternated between, and I try to see which are and which are not being alternated.
One example of alternation I watch for is studying specification vs. working with the product. These two activities feed each other very well. Whereas it is a little crazy to expect that I shall read ALL the spec and memorize it, only then to begin working with the product.]Â
September 3rd, 2006 at 1:25 am
Well, I was going to respond that the alternation point is still self-evident, since most testers I’ve worked with know to switch to different tasks or products in order to keep observation sharp. But I have noticed that when testers are working under a lot of the bad kind of pressure we are less likely to do healthy things like this without prompting. Where could I find more evidence of the importance of a task switch being to its ‘opposite’? (What’s the criteria for opposite? Maybe taking a break is the opposite of working?)
[James' Reply: Alternation behavior is common among testers. But how do you know that most testers you've worked with know to switch to different tasks or products to stay sharp? I suspect you've made casual observations of this. When I observe testers using this instrument, I am trying to make a careful and methodical observation. I'm not satisfied that this behavior is merely in the realm of common sense.
I have found that most people wave vaguely at testers and say things like "this one is pretty good, but that one is very good" without having gathered any systematic observations of the behavior of those testers (none that they seem able to report to me, anyway). For myself, I want a higher standard. This document is a step toward that, I hope. It's an experimental document.
The polarities are not necessarily opposites, but may be. They are in all cases complementary, according to my introspection. I've come to the polarity list (which is certainly not a final or authoritative list) through personal experience and brainstorming with my brother Jonathan. Both of us have had a lot of experience testing and coaching testers on the job, so we believe we have a feel for this. I am interested in suggestions to improve the list.
I know there is hard evidence for the value of alternation in Cognitive Psychology literature. See papers on anchoring effects, framing effects, availability bias, and schema activation. See literature on "de-biasing". There is ample reason to believe that diversifying activities and ideas leads to better searches of a given solution space. More specifically, a complementary activity is one that creates material that can be input to its complement. Example: performing a test can give me information that helps me create a better test. Therefore, designing all the tests at the beginning, and then performing them, without alternating back to design, can be a terrible practice.]
To clarify my comment about the ‘Evolving Work Products’ section, one of the things I would expect to see is a body of empirical data, in various stages of organization/reporting, that details test results - actual responses or measurements from the software under test. And I’m having trouble finding where this would fit into your categories so far. Narrative doesn’t appear to cover it; Test Data you describe as input data, not results. This is the only reason why I see the existing list of categories as preliminary; because it seems to be missing that piece.
[James' Reply: It fits partly under "bugs," but I agree with you that is an oversight. It's not natural for me to think of the test results as an independently evolving thing-- since I think of test results as just a dependent projection of the evolving product and the evolving tests. However, on reflection I think there are situations where we maintain benchmarks or master output files that we have to tweak. So, it could be independent. I will add it in. Thank you!]
September 4th, 2006 at 1:44 am
Under the section “Exploratory Testing Polarities” , Could a point of - Thinking V/s Re-thinking add further value?
I have had an experience of re-thinking on a point during testing, which I had earlier assumed as a well thought point to proceed. It either gave me more confidence to proceed or made me to rethink on something that has already been rethought.
I see that my co-workers too have the experience of - Thinking V/s Re-thinking even for non exploratory testing.
[James' Reply: My first reaction is that thinking and rethinking is another way of saying generation and elaboration. Two things form a polarity, in my parlance, when each one stands on its own, yet benefit from the outcome of the other. Re-thinking doesn't seem to stand on it's own. It's just thinking about the same thing, only more so. ]
September 6th, 2006 at 11:56 am
When evaluating/coaching/diagnosing indiv tester or org for perfomance improvements, do you ever look at “how” they are doing things rather than what they are doing?
[James' Reply: I think I'm always looking at how, to the degree that is possible without using brainscanning equipment. Well, and also to the degree that I have the skill to see how.
How is a very intriguing question. How does the tester know that it's a good idea to re-think the model of the product right now? How do I know that's a good idea, or a bad idea? What are the causes and effects involved? What elements in the environment influence the relationship between cause and effect?
I would say that how and why important thoughts do or do not occur to us is one of the great problems we must solve to expand and deepen the discipline of exploratory testing.]