Maria writes:
A) Your presentation Test Strategy: “What is it? What does it look like?” applies to creating a test strategy for a specific application (I’ve also read Ingrid B. Ottevanger’s article on “A Risk-Based Test Strategy”). How can I apply the idea to an overall test strategy for the company that I’m working for? Is it possible to create a really good test strategy for a company so that it covers several diverse applications? Having difficulties in finding a way to create a non-poorly stated strategy from which we can create an efficient test process it leaves me with another question: “How do we make a clear line between the overall test strategy and the company test process?”
B) The precondition for this activity is unfortunately not the best leaving us with a tight time schedule and very little time to do a thorough work. My concern is that neither the strategy nor the test process will actually be something possible to use, leaving us with as you say “A string of test technique buzzwords.” So how can I argue that the test strategy and the test process are not just two documents that we have to create but it’s the thoughts behind the documents that are important.
Test strategy is an important yet little-described aspect of test methodology. Let me introduce three definitions:
Test Plan: the set of ideas that guide a test project
Test Strategy: the set of ideas that guide test design
Test Logistics: the set of ideas that guide the application of resources to fulfill a test strategy
I find these ideas to be a useful jumping off point. Here are some implications:
- The test plan is the sum of test strategy and test logistics.
- The test plan document does not necessarily contain a test plan. This is because many test plan documents are created by people who are following templates without understanding them, or writing things to please their bosses, without knowing how to fulfill their promises, or simply because it once was a genuine test plan but now is obsolete.
- Conversely, a genuine test plan is not necessarily documented. This is because new ideas may occur to you each day that change how you test. In my career, I have mostly operated without written test plans.
One quick way to think about test strategy is to realize that testing is (usually) a process of constructing an explanation of the status of the product. Therefore, the ideas that should guide our testing are those that relate to the marshalling of evidence for that explanation.
Here’s an example: Let’s say I need to test Inkscape, an open source painting program. The people I work for want this product to be a viable alternative to Adobe Photoshop.
This leads to an overarching question for the testing: “Given the general capabilities of Inkscape, is the program sufficiently reliable and are its capabilities well enough deployed that a serious user would consider that Inkscape is a viable alternative to Photoshop?” This is a question about the status of Inkscape. Answering it is not merely a processing of determining yes or no, because, as a tester, I must supply an explanation that justifies my answer.
Working backwards, I would have to do something like the following:
- Catalog the capabilities of Inkscape and match them to Photoshop.
- Determine the major objectives users might have in using a paint program, as well as various kinds of users.
- Learn about the product. Get a feel for it in terms of its structures, functions, data, and platforms.
- List the general kinds of problems that might occur in the product, based on my knowledge of the technology and the users.
- Decide which parts of the product are more likely to fail and/or are more important. Focus more attention on those areas.
- Determine what kinds of operations I need to do and which systematic observations I need to make in order to detect problems in the product (area by area and capability by capability) and compare it to Photoshop. (Here’s where I would also apply a variety of test techniques.)
- Carry out those test activities and repeat as necessary.
- Consider testability and automation as I go.
In doing these things, I would be gathering the evidence I need to argue for the specific ways in which Inkscape does or does not stand up to Photoshop.
Company-wide Test Strategy
In my way of thinking, a good test strategy is product specific. You can have a generic test strategy, but since you don’t test generic products, but only specific products, it will become better when it is made specific to what you are testing at the moment.
Perhaps what you are talking about is a strategy that relates to what you need for a test lab infrastructure, or for developing the appropriate product-specific test skills? Or perhaps you are thinking of creating materials to aid test leads in producing specific test strategies?
If so, one thing to consider is a risk catalog (aka bug taxonomy). A risk catalog is an outline of the kinds of problems that typically occur in products that use a particular technology. You can create one of these based on your experience testing a product, then reuse it for any other similar product.
Company Test Process
I suggest using the term methodology instead of process. “Process” is the way things happen. “Methodology” is a system of methods. You use a methodology; but you participate in a process. When you have an idea and say, “I think I’ll try doing that” the idea itself is probably a method. When you do it, it becomes a practice (a practice is something that you actually do), and therefore it influences the process.
I use these words carefully because process is a very rich and complex reality that I do not want to oversimplify. For instance, it may be a part of your methodology to create a test plan, but at the same time it may be a genuine part of your process that your test plan document is ignored. “Ignore the test plan document” is not going to be written down in anyone’s methodology, yet it can be an important part of the process, especially if the test plan document is full of bad ideas.
The dividing line between test strategy and test methodology is not hard to find, I think. A test strategy is product specific, and a test methodology is not. Another important element you haven’t mentioned is test skill. Your methodology is useless without skilled testers to apply it.
I would suggest that a more important dividing line for you to consider is the line between skill and method. How much do you rely on skilled people to select the right thing to do next, and how much are you trying to program them with methodology? Many process people get this all mixed up. They treat testers as if they are children, or idiots, trying to dictate solutions to problems instead of letting the testers solve the problems for themselves. What the process people then get is either bad testing, or, hopefully, their methodology is ignored and the testers do a good job anyway.
When I develop a test methodology, as I have done for a number of companies, I focus on training the testers and on creating a systematic training and coaching mechanism. That way the methodology documentation is much thinner and less expensive to maintain.
