Question: How do you stay sharp as a tester?

Shrini writes: How does a good tester keep his testing abilities sharpened all the times. compare it with keep our body fit as we grow old ( walking, jogging and going to Gym, eating healthyfood etc) – what you suggest for keeping “Tester health”? in “fit and sound”? condition?

Testing is analysis and problem solving. Here is what I did, this past week:

  • I solved about 50 problems from the book “Lateral Logic Puzzles” with my son.
  • Paul Jorgensen sent me an exploratory testing challenge, in the form of a spreadsheet with a bug in it. I investigated the bug and wrote a play-by-play description of what I did.
  • I wrote a Perl script to generate some experimental tests.
  • I practiced Sudoku with my Nintendo DS Sudoku game.
  • I analytically solved a conditional probability problem (the taxicab problem) that is often associated with the Representativeness Bias. This was part of working out a testing exercise based on that bias. (Then I tried the new exercise with Michael Bolton.)
  • I read some of a testing book from 1986 that Mike Kelly lent me. I’m trying to characterize the difference between “modern” testing ideas and those from 20 years ago.
  • This morning, I derived the formula for calculating the distance to the horizon based on eye level. It’s been a long time since I did trigonometry, but it was fun rediscovering sines and cosines.
  • I listened to a few hours of lectures from the Teaching Company about Neo-platonism and other philosophical trends of the dark and middle ages.
  • I skimmed several articles, including Knowledge And Software Engineering: A Methodological Framework To Symbiotic Software Process Modeling, and Blooming E-learning: Adapting Bloom’s Taxonomy into the content of e-learning course to promote life long learning through Metacognition, and Third Cybernetic Revolution: Beyond Open to Dialogic System Theories.
    It may not seem like it from the titles, but they have a lot to do with analyzing testing practices and becoming a better tester.
  • I received Pradeep’s Soundararajan’s startlingly incisive answer to the Wine Glass factoring exercise I gave him (“Describe all the dimensions of a wine glass that may be relevant to testing it.”), which helped me see more angles and subtleties to my question. Then I transpected with Michael Bolton as he worked through the same problem.
  • I worked on answers to testing questions submitted by my readers.

As you see, I stay sharp in testing by finding and solving problems, including testing problems; and reading or listening to philosophical ideas that I use to understand testing better; and by trying to help other testers learn, or by watching them learn; and by actually testing.

I’m not in a project, at the moment, for a paying client. If I were, I would be staying sharp by solving problems for my client. I do my best to find excuses to learn new things while working for pay.

When I worked at Apple Computer, I often stole away to the Donut Wheel, across the street, to read about software engineering. When I worked at Borland, I stayed late and worked on test methodology documents and articles. At SmartPatents, I learned Perl and formed my first thoughts about agile test automation.

Some people, and I know you are like this too, Shrini, sharpen themselves no matter what else is going on.

A Question About Test Strategy

Maria writes:

A) Your presentation Test Strategy: “What is it? What does it look like?” applies to creating a test strategy for a specific application (I’ve also read Ingrid B. Ottevanger’s article on “A Risk-Based Test Strategy”). How can I apply the idea to an overall test strategy for the company that I’m working for? Is it possible to create a really good test strategy for a company so that it covers several diverse applications? Having difficulties in finding a way to create a non-poorly stated strategy from which we can create an efficient test process it leaves me with another question: “How do we make a clear line between the overall test strategy and the company test process?”

B) The precondition for this activity is unfortunately not the best leaving us with a tight time schedule and very little time to do a thorough work. My concern is that neither the strategy nor the test process will actually be something possible to use, leaving us with as you say “A string of test technique buzzwords.” So how can I argue that the test strategy and the test process are not just two documents that we have to create but it’s the thoughts behind the documents that are important.

Test strategy is an important yet little-described aspect of test methodology. Let me introduce three definitions:

Test Plan: the set of ideas that guide a test project

Test Strategy: the set of ideas that guide test design

Test Logistics: the set of ideas that guide the application of resources to fulfill a test strategy

I find these ideas to be a useful jumping off point. Here are some implications:

  • The test plan is the sum of test strategy and test logistics.
  • The test plan document does not necessarily contain a test plan. This is because many test plan documents are created by people who are following templates without understanding them, or writing things to please their bosses, without knowing how to fulfill their promises, or simply because it once was a genuine test plan but now is obsolete.
  • Conversely, a genuine test plan is not necessarily documented. This is because new ideas may occur to you each day that change how you test. In my career, I have mostly operated without written test plans.

One quick way to think about test strategy is to realize that testing is (usually) a process of constructing an explanation of the status of the product. Therefore, the ideas that should guide our testing are those that relate to the marshalling of evidence for that explanation.

Here’s an example: Let’s say I need to test Inkscape, an open source painting program. The people I work for want this product to be a viable alternative to Adobe Photoshop.

This leads to an overarching question for the testing: “Given the general capabilities of Inkscape, is the program sufficiently reliable and are its capabilities well enough deployed that a serious user would consider that Inkscape is a viable alternative to Photoshop?” This is a question about the status of Inkscape. Answering it is not merely a processing of determining yes or no, because, as a tester, I must supply an explanation that justifies my answer.

Working backwards, I would have to do something like the following:

  1. Catalog the capabilities of Inkscape and match them to Photoshop.
  2. Determine the major objectives users might have in using a paint program, as well as various kinds of users.
  3. Learn about the product. Get a feel for it in terms of its structures, functions, data, and platforms.
  4. List the general kinds of problems that might occur in the product, based on my knowledge of the technology and the users.
  5. Decide which parts of the product are more likely to fail and/or are more important. Focus more attention on those areas.
  6. Determine what kinds of operations I need to do and which systematic observations I need to make in order to detect problems in the product (area by area and capability by capability) and compare it to Photoshop. (Here’s where I would also apply a variety of test techniques.)
  7. Carry out those test activities and repeat as necessary.
  8. Consider testability and automation as I go.

In doing these things, I would be gathering the evidence I need to argue for the specific ways in which Inkscape does or does not stand up to Photoshop.

Company-wide Test Strategy

In my way of thinking, a good test strategy is product specific. You can have a generic test strategy, but since you don’t test generic products, but only specific products, it will become better when it is made specific to what you are testing at the moment.

Perhaps what you are talking about is a strategy that relates to what you need for a test lab infrastructure, or for developing the appropriate product-specific test skills? Or perhaps you are thinking of creating materials to aid test leads in producing specific test strategies?

If so, one thing to consider is a risk catalog (aka bug taxonomy). A risk catalog is an outline of the kinds of problems that typically occur in products that use a particular technology. You can create one of these based on your experience testing a product, then reuse it for any other similar product.

Company Test Process

I suggest using the term methodology instead of process. “Process” is the way things happen. “Methodology” is a system of methods. You use a methodology; but you participate in a process. When you have an idea and say, “I think I’ll try doing that” the idea itself is probably a method. When you do it, it becomes a practice (a practice is something that you actually do), and therefore it influences the process.

I use these words carefully because process is a very rich and complex reality that I do not want to oversimplify. For instance, it may be a part of your methodology to create a test plan, but at the same time it may be a genuine part of your process that your test plan document is ignored. “Ignore the test plan document” is not going to be written down in anyone’s methodology, yet it can be an important part of the process, especially if the test plan document is full of bad ideas.

The dividing line between test strategy and test methodology is not hard to find, I think. A test strategy is product specific, and a test methodology is not. Another important element you haven’t mentioned is test skill. Your methodology is useless without skilled testers to apply it.

I would suggest that a more important dividing line for you to consider is the line between skill and method. How much do you rely on skilled people to select the right thing to do next, and how much are you trying to program them with methodology? Many process people get this all mixed up. They treat testers as if they are children, or idiots, trying to dictate solutions to problems instead of letting the testers solve the problems for themselves. What the process people then get is either bad testing, or, hopefully, their methodology is ignored and the testers do a good job anyway.

When I develop a test methodology, as I have done for a number of companies, I focus on training the testers and on creating a systematic training and coaching mechanism. That way the methodology documentation is much thinner and less expensive to maintain.

Transpective Dialogs for Learning

One of the techniques I use for my own technical education is to ask someone a question or present them with a problem, then think through the same issue while listening to them work it out. As it proceeds, I ask handfuls of Socratic questions. As I ask each question out loud, I answer it myself silently, and compare my answers with the ones I hear from my counterpart.

It’s not introspection, since I’m not looking solely within myself. It’s also not an examination or an inspection, since my purpose is not to evaluate my partner. So, I coined a new word, which with Google’s help I discovered is not a new word at all: transpection; or transpective dialog. Verb form: transpect.

Transpection basically means to learn by putting yourself in someone else’s place. The transpective dialogs I do are about using someone else’s knowledge, biases, and methods as a counterpoint to my own as I try to solve a problem for myself. As I do this, I generally don’t share my own thoughts, for fear of biasing my partner toward my own way of thinking. The essence of transpection is to get the maximum value out of seeing the world as the other person sees it.

Problems With Transpection

  • Most people feel like they are being interrogated or even tortured (and not in the good senses of those words) when I ask all those critical questions during transpection. Whereas I know what my intentions are, they may not, and sometimes they don’t believe me when I tell them.
  • Some people think I’m judging them, even when I’m not. And sometimes I am also judging them.
  • In one extreme case, I was accused of treating someone like a lab rat. I tried to explain that “lab rat” is just a loaded way of saying “someone from whom I learn” and yes, I did treat her as someone from whom I could learn. She was not mollified.
  • In general, I find I don’t mind when someone does a transpection on me. Actually, I never know for sure if that’s what they’re doing. From my point of view it just looks like they are very interested in what I have to say. I like being listened to. I like to talk. The interrogation aspect rolls off of me, for some reason. Maybe I just like taking tests. This may be why it’s taken me years to realize that everyone does not enjoy being questioned.

Tips for Happier Transpection

Here’s what I have begun to do to help my colleagues and students get excited about doing transpection with me:

  • I distinguish between shallow and deep transpections. With shallow transpection, I don’t ask many questions, the questions are not as sharp, and I share more of my thoughts along the way. Shallow transpection is outwardly identical to that which we call “listening” and is not at all controversial.
  • I explain the transpection process and ask permission before doing a deep transpection.
  • I avoid doing deep transpection on people who are not my students or close colleagues.
  • I invite them to do transpection on me.
  • I watch their reactions and if they seem to get irritated, I disclose some or all of my own ideas and feelings about the situation I’m inquiring about. I will also disclose the specific motivations behind each question I ask.
  • I talk about transpection as part of a general philosophy of sophisticated team learning. To do this with me is to participate in an exciting and vital process.
  • I make a point of listing the things I’ve learned and thanking my partner for helping me learn from them.

Here are some quotes I came across while Googling about this:

Milton J. Bennett: “[Transpection is] the ability to imagine oneself in a role within the context of a different culture”

Yasuhiko Genku Kimura: “Transpection includes but transcends extrospection and introspection.”

Silvia Teresita Acuña and Graciela Elisa Barchini: “This process consists in getting into the “headâ€? of another person. This means trying to think like the other person (temporarily, one person “believes and feelsâ€? everything that the other person “believes and feelsâ€?). This process of “assumingâ€? the thoughts of another person is not easy and it will take people who have never tried it a long time.”


Another common problem in transpection just occurred to me while reading some of the comments. When somebody I’m transpecting says something that reveals a new insight for me, I tend to get excited, and I follow up with questions that I’m told sound especially sharp and even angry. Actually I’m very happy, as a shark is happy to find a tasty sea lion. The sea lion might get the impression that the shark is angry, but he’s really not.

This is a critical moment, because just when I’m learning something profound, my partner may take his brain and go home. I’m still learning how to manage this. For now, I’m consciously forcing myself to say happy encouraging things while this is happening, instead of getting carried away by the content and implications of the new insight and asking only content-oriented questions. It’s not a foolproof solution, though. I’ve found that people sometimes don’t believe me when I say I’m learning something from them. Sometimes they think I’m mocking them. It’s an imperfectly solved problem, in my practice so far.

My struggle is partly that a lot of transpection feels, when I initiate it, like a personal learning process, whereas it is actually, of course, a social process.

Question: Tester’s Freedom of Thought

Subha asks:
A tester is usually bound by the constraints of specifications when he does functional testing. But what about usability? How much should the tester’s imagination be allowed to flow?

Hello Subha,

Read carefully– this is important:

The specification does not bind you, as a tester. The specification provokes you. In fact, the spec, the product, the things people say on the project– all of it is provocative to the tester. It might be where we start, but not where we end. If the product behaves in some important way (important either positively or negatively), then it is generally the role of the tester to test it, even if there is nothing in the spec about that behavior.

Doing testing well requires a great deal of imagination.

I think a tester is actually bound by six things that come to mind as I write this: the mission, project culture, the paticular constraints of the project, and the skill and knowledge of the tester. All of these except that last one are to some extent negotiable:

Mission: This is the problem that your clients want you to solve for them; the outcome they want you to achieve. If you don’t honor your mission, you will not gain credibility or retain respect. Be sure that you negotiate a mission you are capable of fulfilling; and be sure that the story of your testing features the mission as its primary plot point. Is usability testing a part of your mission?

Project Culture: The other people on your project have expectations about what you will or will not do. These bind you. You can challenge those expectations and suggest alternatives, but you have to be careful about that. If usability is part of your mission, what methods of usability testing are acceptable or expected within your organization?

Project Constraints: On your project, you don’t have all the time and money to do everything you might find useful or interesting. You may need to find inexpensive ways to do the testing that your strategy calls for. You may need to acquire special tools, or use tools that don’t do everything you wish they did. What kind of usability testing is it even possible to do on your project?

Tester Skill & Knowledge: Even if you were granted permission and resources to do everything you want to do, you would still be limited to the things that you know how to do. If you want to do testing well, you need enough command of testing practices and tools to make that possible. One common problem with testers is that they don’t do enough to educate themselves. Do you know how to do usability testing?

I realize that this is not a detailed answer to your question. What I’m trying to do is frame a way for you to think the issue through for yourself.

Ethical Standards: A tester is bound by ethical standards not to, for instance, lie about the results of the tests, or to misrepresent his ability to do the work. Are you suggesting usability testing for selfish reasons, or do you really believe it will help your client?

Legal Standards: A tester is bound by legal standards. In some cases, there are laws, such as Sarbanes-Oxley or HIPAA, which guide how you must test. Is there any legal reason why you must or must not perform usability testing?