Deep Analysis of Steps in a Scripted Test
Recently, Michael Bolton and I had a conversation, which he recorded, about the deep nature and meaning of a scripted test procedure. It’s an MP3 file that is a bit more than an hour long.
If you think that scripting a test is a simple matter of “writing down what the tester should do”, then I would urge you to listen to Michael and I deconstruct a simple example of a test script.
We take the first step of a hypothetical test and analyze it. That step is:
“1. Reboot the test system.”
What does this mean? What should the tester do? Consider these questions for a moment before listening to the MP3.
April 23rd, 2007 at 8:37 pm
This discussion reminds me of some exchanges I’ve had with test script authors when I am asked to automate their test scripts. There is a great deal of ambiguity in written manul test scripts and procedures. Often things that the author thinks are not ambiguous can have numerous interpretations. And as one automating or executing someone else’s test scripts, I do not have all the context information that the script author considered when the wrote down the test steps.
I can’t code automated execution of ambiguous test steps. (And I often question how the manual testers execute ambiguous scripts.) The process of automating scripted tests requires removing the ambiguity and putting it into executable code. For automation, the line often needs to be drawn at a point with much more detail than may be required for manual testing. The need to nail down the ambiguity makes it easy to automate something that differs from that the script author thought they requested.
When manual testers take those same ambiguous scripts, they are going to interpret them differently each time the sciprts are executed. The scripted automation is going to be much more consistent than the manual tester. Which is better? It depends on the context.
Numerous interpretations of the ambiguity may make testing better. Or the wrong interpretation of a test step may not exercise the system as the script writer intended. And automating the wrong interpretation can have negative impact into perpetuity.
How do we teach script writers to lock down those things that need to be locked down? When I ask questions to try to nail down some of the ambiguity, I get the impression that the script writers think I’m trying to make simple things difficult.
April 24th, 2007 at 5:01 am
I listened to the audio but I did something different than just listening to it.
I knew this was going to be a great discussion and hence I paused the audio whenever a question was being asked or a point being made by James or Michael. After a pause, I took notes ( my answers to those questions, the questions I had, ideas I got, points that were necessary and helpful to note down, in my opinion) and I enjoyed a very different learning experience. Merely listening and thinking would have been less fun and learning. Maybe this way worked great for me.
And of course, I am writing this comment without rebooting my PC
and yes James/Michael, “it appears to be so”.
April 24th, 2007 at 12:17 pm
>I can’t code automated execution of ambiguous test steps.
Sure you can. You do it all the time. It’s just that you either perceive no ambiguity, or you agree to accept the level of ambiguity that you perceive. I see authors and presenters at conferences constantly trying to reify ambiguity, but ambiguity isn’t a thing; it’s a relationship between a person and some expression of some idea. It’s a cognitive, subjective thing. (By the way; when some presenter advocates “unambiguous requirements”, I like to ask them, “What do you mean by ‘unambiguous’?”
>(And I often question how the manual testers execute ambiguous scripts.)
All the time; same answer as above.
>The process of automating scripted tests requires removing the ambiguity and putting it into executable code.
Hmm. I’d be careful and suggest that the process of automating scripted tests requires translating some question about some perceived requirement into executable code, such that the test code sufficiently models the testing question.
>Numerous interpretations of the ambiguity may make testing better. Or the wrong interpretation of a test step may not exercise the system as the script writer intended. And automating the wrong interpretation can have negative impact into perpetuity.
>How do we teach script writers to lock down those things that need to be locked down? When I ask questions to try to nail down some of the ambiguity, I get the impression that the script writers think I’m trying to make simple things difficult.
Here’s how I think we could go about it. I’ve ranked these–the most urgent first and the most important last.
The first step is to acknowledge, from the beginning and repeatedly after that, that you’re trying to help–and that you want to avoid the risk of misinterpreting an idea.
The second is to remind them that, as a tester, it’s your job to consider questions and possibilities that other people don’t consider. You might like to, on occasion, recognize with the other person that it’s is a peculiar and potentially funny occupational hazard.
The third is to try to solve the problem by ways other than locking them down. (Hands up, everyone who likes to be locked down.) Consider things like culture, common language, shared idioms, mentoring, training, observation, participation, collaboration,… These might suggest alternatives ways of understanding one another.
Fourth, and most importantly: when you write a test script, either for automation or (ugh!) for human testers, it’s to further some purpose, to answer some question about the product. Maybe if you understand the question that’s being asked, to whom it matters, and why it matters, clarity of each and every little individual step doesn’t matter. What matters is the fundamental question of testing–”is there a problem here?”–closely followed by, “if I’m automating a test, what problem am I hoping to identify? How will this script help me to identify that problem? What steps would the script need to perform to get to that identification? What problems might I miss?” At that point, you-the-toolsmith can write the script without painful interrogation of some poor soul who might not know how to break the task down in a way that’s entirely helpful to you. Making that translation from risk and test ideas to scripts expertly, without the need for hand-holding, is the real centre of skilled toolsmithing to me.
—Michael B.
April 25th, 2007 at 1:20 am
@Michael
>>I can’t code automated execution of ambiguous test steps.
>Sure you can. You do it all the time. It’s just that you either perceive no ambiguity, or you agree to accept the level of ambiguity that you perceive.
I agree. I can automate ambiguous test steps by translating them into executable code — just as a manual tester translates them into executable test steps. (Its just that if I code it into automation wrong, it is likely to be executed wrong over and over.) What I meant to say is that to automate something that I perceive to be ambiguous (or something that I fail to recognize as ambiguous), I have to give a computer very specific steps to execute. I can’t give a computer an ambiguous instruction in an automation script. I can, however, automate specific steps for specific interpretations of that test step.
>>The process of automating scripted tests requires removing the ambiguity and putting it into executable code.
>Hmm. I’d be careful and suggest that the process of automating scripted tests requires translating some question about some perceived requirement into executable code, such that the test code sufficiently models the testing question.
I think that’s what I intended to convey.
And thanks again for some good advice for helping determine what questions are to be answered by automation without painful interrogation. I want to collaborate with test script writers, not interrogate them.
Ben
April 25th, 2007 at 4:36 am
This reminds me of an experience. I’ve got the test step “reboot the computer� in one of numerous (about 5 person-days to run) manual test scripts. The only test script included that step (related to recovery of NT services). I remember how I found a regression (!) defect executing this script - exactly because I did reboot. I remember I guessed almost no risk of server functionality to be broken. So my mind resisted, but script insisted to run that test and I did and I found the problem - the bad one.
Maybe I’m so lucky to work in a project where we could only write down those steps that we (testers) believe are associated with some risk. And only provide as much details as required to address the risk?! From what I hear and read here You scare me – I’m afraid to get into another project/company where it is not a case!!!
In my example I know that the risk is about NT services startup, so I don’t care how the computer is reboot, but only that it is reboot. I never write test step 1 login using user name A into our test system (it is obvious, isn’t it?) unless there is step X login with a user name B and make sure you have an item send from A during the step X-Y.
We write them in a way best suited for manual execution of a person familiar with a product. We automate whatever needs to be automated ourselves and we never directly automate exactly the same tests as designed for manual tests, but that’s an another story.
[James' Reply: I think what Michael and I are trying to show is that this is not an easy problem to solve. If you give me a script to follow, I may well understand your script differently from how you intended. You are always making a tradeoff decision when you document a test procedure, and your tradeoff may or may not pay off. I'm impatient with people sometimes, because I've been doing testing and witnessing testing for twenty years, and in that time I've seen so many people who thought test scripting was easy get burned. So, for me to hear someone imply that this is just a simple matter of putting obvious steps in an obvious order is like hearing a teenager say that he can drive a car just fine drinking after a liter of whiskey.]
April 25th, 2007 at 9:59 am
Innocently looking, pretty straight forward steps like “Reboot the system” and “test windows media player” on a Windows XP embedded” etc, can have serious hidden sub steps and possible variations under them.
One of the perils of scripted testing is that there is no time and least tolerance for probing and exploring questions (especially ones that might look embarrassing, late questions). The notion here is that questions delay the completion and hurt testing mission that is mostly not explicitly stated.
[James' Reply: I would say that the peril relates to the way most people think of scripted testing, and it's a peril in the concept of scripting, but it is not necessarily a peril with a hybrid scripted/exploratory approach. A test can be scripted without being SO scripted that the tester can't change it or question it.]
I blogged about two extreme design (rather documentation) styles of manual test scripts here – though on a different context (ways to make automation difficult).
It is often seen that scripted tests are written and documented with no focus or linkage to “ultimate aim of testing” - they are written in either “context free” or specific context situations that invariably differ from actual testing missions and contexts in which tests are actually is applied. This leads to that fact that tests (scripted) ones that are meant to be reused across testing tasks with differing objectives - making a poor choice and re-usability stuff.
This can also be viewed as an example of Goal displacement as you referred in some discussion on Software testing yahoo group. In most of the cases I see that original goal of testing via test scripts was displaced to make the task of test execution easier (detailing the steps to the “death�)
[James' Reply: It may make the task of execution easier in the sense of being able to say "I did what was written down", but not easier in the sense of "I performed a good test." The goodness of a powerful and resilient test requires the cognitive involvement of a tester. Only a weak test can be performed by a thoughtless drone. An interesting thing about humans is that when you ask them not to think much, most of them will not think enough. It's easy to turn off our brains. Thinking well is a risky and difficult proposition.]
People assume that test scripts are like the java code – “Write once (or possible think once) and execute tests anywhere (context and/or testing mission)�
Here is where exploratory tests fit nicely in the scheme of things — for every instance of test design - the information and format of test cases can be perfectly aligned with testing mission. In fact very first question in a exploratory design would be “what is testing mission - what tests are trying to achieve? Are the objectives aligned with project sponsors and stakeholders?”
Can you imagine asking such question in a scripted testing? When done that way, a test might be tied to specific testing mission. That would be a good thing as we now know what that test is for.
[James' Reply: In a hybrid scripted/exploratory test activity, I would such ask questions.]
Note that when you say “Don’t not think about those variations in tests” and “These tests are not designed to expose potential problems” – indicates clear “disconnect between” testing mission and the test case. Worst part here is that such a “disconnectâ€? was *accidentally* discovered. Such a “disconnectâ€? undermines the value and effectiveness of tests. Unless some one asks question (unintentionally) and discovers the “disconnectâ€?, we may never know a “disconnectâ€? existed. That seems to be typical with scripted tests.
[James' Reply: The disconnect was not accidentally discovered. Listen again. Actually I was pleased that Michael's systematic questioning exposed this trap. Michael approached the script with his eyes open and alert for misunderstandings.]
Now, consider typical situation test is designed by one and executed by the other (several times after first run) and there is some time gap. What all can happen here –
1. Original test might have been designed with insufficient information
2. Test might have been tied to a different objective or mission
3. Due to time gap, test might have become obsolete and quite lot other things that expose issues of not clarifying testing mission and designing tests on that …
So two key take aways for me from this post are – One, scripted tests often cause goal displacement of testing mission to something that appear to help the situation from some one’s view point (mostly manager’s), one that is not explicitly known and cared about.
Second (more of my interpretation), Too little or too much details in scripts can hurt some ones mission in testing – like making automation difficult.
Shrini
[James' Reply: Yes, and I would add one more: Even the simplest instructions require interpretation. Interpretation is context-specific skill. You can laugh it off or ignore this, but if you do you will periodically discover (if you are paying attention) that the testing being done isn't the testing that you specified. Humans are not programmable in the same way that computers are.]
May 11th, 2007 at 9:55 am
>So, for me to hear someone imply that this is just a simple matter of putting >obvious steps in an obvious order is like hearing a teenager say that he can >drive a car just fine drinking after a liter of whiskey.]
Interestingly with regards to the discussion on ambiguity, it’s true also that the very nature of natural language is ambiguous. For example in James’s comment above the term Liter has no meaning to me because I’m English, however I surmise that he means litre through experience, although I could have also thought that he meant litter of whiskey and that a litter was some unit of measurement. The context of words is very important to the meaning. Many of the issues in understanding our natural language occur in the field of understanding the context in which the words are spoken (or written). I guess my comment is really that without experience and contextual understanding many test steps are entirely useless.
You’ll also note that I’ve used an additional s after the apostrophe when referring to James’s name. That’s because my personal preference for possession when names end in an s is to put apostrophe s, which is as valid as putting just an apostrophe depending on who you talk to - just an interesting aside :-).
[James' Reply: The whole premise of testing is that mistakes are easy to make in this complex and ambiguous world.]Â