Tools For Recording Exploratory Testing
I often criticize pre-scripted testing. It’s not a fundamentally bad idea, but it’s strangely over-hyped. Many writers in the field act as if they suffer from anterograde amnesia– like in the movie Memento, where the guy can’t form new memories and must write everything down or he will instantly forget it. They claim an unscripted test will lead to some kind of total loss of any recordable test progress.
This is ridiculous. For instance, consider:
- Do a crossword puzzle. Now erase all the whole thing. Do it again. Aren’t you a LOT faster the second time through?
- Solve a jigsaw puzzle with, say, 500 pieces. Now jumble it up and solve it again. Aren’t you faster now?
- Learn a new game. Say, checkers or poker. Notice how you get better and more sure of yourself as you play?
These improvement are happening not because you are recording every little event. They occur because your mind absorbs information and experience, and is able to recall and reuse that information.
You can be successful with unscripted testing because your brain is a lot better than you think it is. Give your brain a chance.
Recording Exploratory Tests
Still, it may be very useful to record what happens when you test, whether you are following a script or not. Here are some reasons why:
- You can’t look at everything at once. Even if you try very hard.
- When you get excited about or interested in one thing, you may miss other things.
- When you get tired or bored, you attention might wane.
- You might not pay attention to a particular event or factor that only later turns out to be important.
- Some things are labor-intensive to reproduce (think of a 20-minute installation process).
- You might need a vivid demonstration of something that happened, when words fail to express it.
- Some things that matter aren’t visible through the normal user interface.
- You might have experienced something only as the result of an unusual or intricate series of actions that are hard to recall.
- If, months later, it emerges that a bug escaped into the field, it can be very helpful to have detailed records to consult, as part of the post mortem.
- There might be some external documentation requirement.
I’ve been using various tools to make these records. These include Spector, from Spectorsoft, or BBTestAssistant from Blueberry Consultants, or for the truly budget impaired, Wink.
Even though I can get along without such tools, if I really did have to do a crossword puzzle twice, or was learning a complicated new game, detailed records might come in very handy.
Check Out TestExplorer
Recently, I’ve become excited about a tool from Sirius SQA called TestExplorer. TestExplorer, as far as I know, is the first commercial tool to support Session-based Test Management, which is a formalized approach to managing and reporting on exploratory testing. TestExplorer does some cool things, including:
- keystroke logging
- desktop video w/audio
- file monitoring
- resource monitoring
- registry monitoring
- system information
- test case documentation support
- report generation
- issue management
- test session metrics
What I like most about the tool is that David Gilbert, who wrote it, has been amazingly responsive to feature requests and concerns. Commercial tool developers generally ignore their users, so that’s remarkable!
I have personally tested software using TestExplorer. I’ve watched David test using the tool, too. I recommend it. Check it out.
David is also the first person other than my own brother to earn a BCRIT certification from me. BCRIT (Bach Certified Rapid Tester) is my attempt to create a tester certification program that has integrity. It’s still experimental, but it means that I have personally witnessed the certified person exhibit tactial rapid testing skills while testing something challenging, on more than one occasion, and I believe “that guy can test!”
More on that in another post.
November 9th, 2006 at 10:36 pm
OK! This does it for me.
I feel totally embarrassed by not proving any public credits for David Gilbert and his TestExplorer tool any sooner.
I have been one of the betatesters for TestExplorer. I don’t have a good feel for how many testers participated in betatesting, but I’d like to think I found at least 1 bug.
I only wish I could have done more testing for David Gilbert. Being a betatester for David Gilbert is fun because you receive immediate response on all your questions, proposals and concerns. (Remember my testing while I was in Switzerland David!).David won’t make you any promises that he can’t keep. I very much like his honesty. (You can always contact me for more testing David!)
I have been introduced to the tool about 2 years ago (if I remember well) at the STAREast conference. I am sure David will correct me if I am wrong.
TestExplorer wasn’t at all what it looks like now, but from the start I believed in the concept.
Whenever I hear the words “test automation” all my defense shields go up. Over the first 10 years of my career I have been involved in test automation, record-playback that is. (The kind of tools like WinRunner ). I have never believed in the record-playback concept. I have been hurt by it every time that used it.
TestExplorer is different! TestExplorer allows a tester to stay focused on his thinking patterns, without having to worry about documentation.
I see opportunities for this tool outside of the testing field as well. Training is the first thing that comes to mind. It is great for creation of user guides as well.
I believe that TestExplorer could even be used as a testing tool in an FDA regulated market.
I recently moved my self in the position of project manager. I have convinced one other person to apply SBTM during an evaluation project. SBTM will be applied on a larger scale in the project that I am managing.
I will defenitly use TestExplorer. I’ll report to both David and you about my experience.
I am not sure what I like the more: the tool or its owner.
Like you stated James, I know David as a very responsive person. I always received immediate replies to my questions, proposals and concerns.
Looking forward for more …
Erwin Van Trier.
November 10th, 2006 at 8:25 am
About pre-scripted testing…
For me, a good reason to have pre-scripted testing in some areas is the ability to have different people execute the tests in a more predictable way and shorter time than would otherwise be possible. I cannot always count on having the tester with the most knowledge and previous testing experience in a certain area carry out “all” testing there. Sometimes an optimal match between assignment and assignee is sacrificed for flexibility in planning.
It’s dangerous to rely just on “good stuff” inside someone’s head. To be a learning organization, some has to go on paper (or the web… or… well you get the idea
It doesn’t have to be a step by step instruction, rather a check list, or a list of risks with at least some ideas on how to find bugs related to them.
That said - it’s extremely important not to limit the tester’s motivation to think freely around a test assignment even if a part of it is scripted.
/Henrik
[James’ Reply: I certainly hear this a lot, Henrik, but I think it’s just a rationalization. Why? Because it takes no account of the relative importance of the different variables. You could argue, for instance, that it’s a good thing to drive a specialized race car everywhere you go, instead of an ordinary family car. You could justify this because a race car “goes faster than is possible with a normal car.” But we know this is absurd, because a race car is very expensive, lacks important features, and an ordinary family car happens to go fast enough for almost all situations.
You can point to lots of theoretical advantages that race car may have, and you can point to theoretical advantages of pre-scripted tests. However, those advantages are based on problems that most people don’t actually have in their projects, while ignoring some important problems that nearly everyone does have on their projects. A problem I am constantly facing is that I have a LOT to test in a SHORT period of time. With exploratory testing, I can cover something on the order of ten times as much testing space in the same time with the same resources. That is nothing to casually dismiss!
Many managers seem to be worried that they can’t control their test projects with pre-scripting them. I consider this to be evidence (usually) that the test manager needs to be trained. If your testers aren’t competent to test without a script, that is evidence that your testers need to be trained. We know how to do this training. We know how to do competent and well-controlled exploratory testing. By “we” I mean testers who read the articles and books that are readily available to teach this stuff.
You say it’s dangerous to rely on what’s in a tester’s head? Well, how about what’s in the heads of your executives, or your programmers? Why aren’t you clamoring for your CEO to “script” his work each day? Why not force policemen and firemen to script each of their moves before they make them? Perhaps parents should file scripts they strictly follow in their dealings with their children? The truth of human social and intellectual life is that we do not limit ourselves to what we are able to describe verbally. I could just as easily– in fact, more easily– argue that it is more dangerous to focus on written artifiacts instead of human skill as the primary vehicle of success.
If all you are saying is that the mechanism of paper and pen not be totally discarded as a tool for intellectual work, then, well, nobody is arguing that it should be discarded. But I see no reason to pre-judge which tools must be used. It’s a situational decision.]
November 10th, 2006 at 11:15 am
I have started my professional carreer as a SW tester for a telecom company, back in 1991. I have worked with the most brilliant testers I have ever met. (One day I hope to be as good as them)
Within the next 11 years I don’t remember anyone of use to have used test cases with a pre-defined expected result.
We had a very successful and renowned test team (and by the way … none of these testers were certified).
Now I work for a company that is heavily regulated by the FDA. I am no longer in a testing position, but I do verify the testing documentation. For the FDA it is mandatory to use pre-scripted expected results. (if someone can convince me otherwise I’d be very happy to talk about it). Out of all the test documentation I have reviewed over the past 5 years, I believe that only 10% (very rough estimate, no data to back this) of the executed test cases actually discovered a problem.
Out of all the problems found, 8% are related to a problem with the pre-scripted expected result itself (”Oh, it should have stated 3GB instead of 2GB”).
This leads me to believe that A) we don’t find important problems or B)the important problems that are found have not been discovered by a pre-scripted expected result.
Actually, I have found a statement in a Glenford-Myers book that considers the expected result to be an elementary aspect of testing. I have had discussions with Jon Bach about this, because I don’t believe in the statement. I even submitted a lightning talk for STAREast about the topic. My talk got selected, but my trip got cancelled - bummer.
Some people have tried to argue with me that some expected results are retrospective.
I consider a retrospective expected result to be a contradiction in terms. (more likely to be used in the fortune telling business)
Wikipedia: An expectation, which is a belief that is centered on the future, may or may not be realistic.
Websters: Future prospects
I believe that the use of pre-scripted expected results is more harmful than useful (I might be able to find some exceptions)
November 10th, 2006 at 11:16 am
Henrik — Well, to be honest with you, I agree completely. And, James, to be honest with you, I agree completely. And THAT is exactly why TestExplorer exists in the first place.
One of the key, distinguishing features of our tool is that it documents a testing process in as much or as little detail as you like, either based on watching you freely test, or by writing out the test in advance, or some blend of the two. But at replay time, it never, Never, NEVER dictates what the test or tester does…the TESTING function is ALWAYS in the hands, and brain, of the tester. TestExplorer simply provides guidance and reference to what the intent and history of a given test case was. So it meets both of your goals, if I understand you both correctly. Now, if what you need is very strict adherance to that prewritten script…then, you simply instruct your testers appropriately…”Do Not Deviate…this is important!”…but still, even within that, if something bizarre happens, the tool still collects and records for posterity all of the information, so that you can say with authority, “We did NOT deviate, and look what happened anyway!”.
We believe this is the futue of testing. We believe automation as we know it will fundamentally change, to be a process that facilitates skilled testers, not attempt to replace them. And we have many more new and exciting ideas we hope to bring to market to promote this belief in the future.
More on that later!
Sincerely,
David
November 10th, 2006 at 11:19 am
Erwin — working with you is always a pleasure. While no one likes seeing bugs get through the development process, I must say it is always nice to send you a new build, and watch the immediate collection of emails begin to roll in. While there are other beta testers, you eclipse them in your passion to find new bugs…and that is what makes a good tester, beta or otherwise. And not because you want to embarass me, but becasue you are excited about what you are doing.
And as my previous post indicates, stand by…there is more on the way.
Sincerely,
David
November 13th, 2006 at 9:23 am
Some more thoughts…
“Because it takes no account of the relative importance of the different variables”
I don’t understand this argument, and certainly not the analogy with the race car. I tried to say that sometimes you need a race car, and sometimes you need another type of car.
[James’ Reply: Read your own reply, Henrik. You didn’t say “sometimes you need X and sometimes you need Y.” That’s not at all what you wrote. If you would like to make a statement like that, however, I would appreciate hearing it. Instead, you wrote in flat, categorical terms about a “good reason to have pre-scripted testing”. I don’t yet see that it is a good reason. It sounds like an assertion, floating in the air, of a kind that is easy to make unless the variables and causalities don’t break down the way you imply that they do.]
“However, those advantages are based on problems that most people don’t actually have in their projects, while ignoring some important problems that nearly everyone does have on their projects.”
Hmm well, the reason that I wrote my comment was to inform you, and anyone else that might be interested, that in my projects, my reality, my context, I actually have a situation where I cannot come to the conclusion that pre-scripted testing is without value. I certainly don’t ignore other’s problems but my comment is about _my_ problems and I quite clearly state why pre-scripted testing helps me to solve those. Is it really compatible with your context driven school to dismiss my comment based on “most people” not having these problems?
[James’ Reply: If you don’t want me to dismiss your comments, then identify your context and your reasoning. What you have described is a conclusion you have drawn based on an unspecified context and an assertion about a dynamic which I am not prepared to accept without more evidence. The reason I am not prepared to accept it is that I have experience of my own which seems to contradict your assertions.]
“A problem I am constantly facing is that I have a LOT to test in a SHORT period of time.”
Yeah, that’s certainly one of my problems as well. Right up there with for example:
- Being able to have move a person to a new task (project) and having him benefit from the work/analysis etc done previously on the same or a similar task.
- Assigning test work to a newly graduated person or someone just switching to testing and still get valuable results quickly. (We have not automated all such tasks yet…)
[James’ Reply: You think that pre-scripting helps in those situations? I don’t find that it does. Do you want to say more about why you think it does? I agree that many people say that it does help. I believe most say that because they don’t look closely at what is happening. They don’t look closely at the quality of the testing that they get, and they don’t look closely at the quality of testing that you get from alternative approaches. Often, they assume that there ARE no alternatives.
It is possible to get good value from pre-scripted testing. However, it is not a slam-dunk. It is neither self-evident, nor straightforward, how to achieve good testing using a pre-scripted approach.]
“You say it’s dangerous to rely on what’s in a tester’s head?”
No, I actually didn’t say that. I said “It’s dangerous to rely just on ‘good stuff’ inside someone’s head”. (It may be a language problem since English obviously isn’t my first language) By that I mean that a (my) test organisation benefits from having some of the tests scripted since it makes it highly probable, if you have instructed and motivated your testers, that these tests are run in a way that will find certain kinds of bugs. I’m not arguing that this should be the only way!! But, when it comes to high risk functionality, such as remote firmware updates risking to destroy a device, I like knowing that all the state transitions (of whatever) have been tested. I will also use exploratory style testing to complement this and find new areas to test around.
[James’ Reply: You seem to think that scripting is a reasonable way to assure test coverage. I find that it is more likely to be a way to prevent test coverage. You seem to think that skilled testers are unreliable. I find them far more reliable then unskilled people reading instructions. You speak of probability, but what evidence can you offer that there is a greater probability of getting what you want through procedural scripting?]
“We know how to do competent and well-controlled exploratory testing. By “we” I mean testers who read the articles and books that are readily available to teach this stuff.”
I have attended conferences, a full day’s tutorial on Exploratory testing held by James Lyndsay. I have listened to a couple of your talks, including the one at SAST in Sweden last year. I’ve read endless amounts of books and articles in Better Software, on Sticky Minds, and on the different home pages belonging to you and others in this community. I have practiced all of this during the build-up of a test team of ~10 people. I would say that I have a fairly good understanding of testing and the problems associated with this discipline. But in many aspects I’m just in the beginning of learning.
My conclusion so far is that Exploratory testing is a superb complement to other approaches. I find it most valuable when trying to figure out how to attack a new functionality. Also, it’s great to use when time is extremely limited and the testing approaches smoke testing, especially when time has not allowed for any test case construction ahead of time.
You may well be the person that convinces me that your type of car is the car to beat them all. You’re reasonably close actually. But, you will not do it with answers of the kind you made to my previous comment.
[James’ Reply: It is neither my duty nor my pleasure to flatter you. I’m reporting to you that I don’t see a foundation for your statements. What you choose to do with that information is your business.]
“Why aren’t you clamoring for your CEO to “script” his work each day? Why not force policemen and firemen to script each of their moves before they make them?”
I’m not “clamouring” for anything… And I certainly would hope that some of the activities of both CEOs and policemen are “scripted”. I cannot imagine that a CEO doesn’t follow a more or less step-by-step approach for economic reporting for example. A policeman would probably have a nice set of instructions for tasks such as cleaning his weapon, filing a request of some sort, or even for negotiating in a threatening situation. But… I really don’t know for sure.
“If all you are saying is that the mechanism of paper and pen not be totally discarded as a tool for intellectual work”
Do you actually think that’s what I’m saying?
[James’ Reply: I’m not sure what you are saying. That’s why I said “if”. I am sure that you have not yet said enough to suit me.]
You’ve said about pre-scripted testing that “It’s not a fundamentally bad idea”. When do you think it should be used?
/Henrik
[James’ Reply: That question requires about 5,000 words to begin to answer. I suggest you read “The Social Life of Information” and “Things that Make Us Smart”, first. Then the conversation will be a lot easier for both of us.]