Monthly Archives: February 2006

Exploratory Testing Research

Posted on by
1

Good research on testing is hard to find. Why? One reason is that testing does not belong to the field of Computer Science. I mean, sure, some of it does. There is some value to describing and testing an algorithm to efficiently cover a directed graph. But covering directed graphs is not my problem, most of the time. Most of the time, my problem is how to work with other people to simplify a complex world. Most of the time, the testing problem is an exploration and modeling problem within a socially distributed cognitive system. Whew! Whatever that is, it ain’t Computer Science.

Therefore, I am delighted to present two excellent examples of rigorous scientific research into exploratory testing– both of them coming from the field of Cognitive Science.

  1. Jerry Weinberg’s 1965 Doctoral ThesisHere, Jerry runs an experiment to determine strategies people use when trying to comprehend a pattern of behavior in a system. In this case, the system is a set of symbols that keep changing, and the task is to predict the symbols that will come next. By observing the pattern of prediction made by his test subjects, Jerry is able to draw inferences about the evolution of their mental models of the system.The upshot is this: tom some extent it is possible to see how testers think while they are thinking. I use this principle to evaluate testers and coach them to think better.
  2. Collaborative Discovery in a Scientific DomainThis paper by Takeshi Okada and Herbert Simon is fantastic!They study how pairs of scientists, working together, design and conduct experiments to discover a scientific principle. This is EXACTLY the same thought process used by testers to investigate the behavior of systems.Notice how Okada and Simon collect information about the thought processes of their subjects. It’s very much like Weinberg’s approach, and shows again that it is possible to draw solid inferences and make interesting distinctions about the thought processes of testers.This is important stuff, because we need to make the case that exploratory testing is a rich activity that can be observed, evaluated, and also systematically taught and improved. These two papers deal with the observation and evaluation part, but I think they suggest ways to teach and improve.

Surprise Heuristic

Posted on by
1

At the recent Workshop on Training Software Testers, Morven Gentleman showed us a chart of some test results. I was surprised to see a certain pattern in the results. I began to think of new and better tests to probe the phenomenon.

Morven told us that the tester who produced that chart did not see anything strange about it. This intrigued me. Why did Morven and I see something worth investigation when the tester did not?

Then I stopped myself and tried to discover my own thought process on this. A few minutes later this exploratory testing heuristic came to mind:

I MAKE AN OBSERVATION DURING A TEST…

1. I experience surprise associated with a pattern within the observation.

That triggers REFLECTION about PLAUSIBILITY…

2. The pattern seems implausible relative to my current model of the phenomenon.

That triggers REFLECTION about RISK…

3. I can bring to mind a risk associated with that implausible pattern.

That triggers REFLECTION on MAGNITUDE OF RISK…

4. The risk seems important.

That triggers TEST REDESIGN…

Now, I don’t really know if this is my thought process, but it’s a pattern I might be able to use to explain to new testers how surprise can be a test tool.

Dead Bee Heuristic

Posted on by
1

Have you ever had a software problem that disappeared even when you did nothing to correct it? Or have you ever fixed a bug by doing something that seems as if it shouldn’t have fixed anything?

Whenever that happens to me, I A) remain wary, and B) remove the fix so that by seeing the problem again I have additional evidence that the “fix” was truly the fix. I call this is the dead bee heuristic, because if there’s a bee in my living room, I don’t want it to mysteriously disappear, I want to see it fly out my window or be dead on my floor.

This applies to testing situations, too. If I change a data file and see that it no longer crashes the application I’m testing, the next thing I do is change it back again so I can see the crash one more time.

“Say, was you ever bit by a dead bee?…You know, you got to be careful of dead bees if you’re goin’ around barefooted, ’cause if you step on them they can sting you just as bad as if they was alive, especially if they was kind of mad when they got killed. I bet I been bit a hundred times that way.” — Walter Brennan as “Eddie” in To Have and Have Not

And always bear in mind that killing the “bee” may not have solved the real problem, or may have created new problems.

Peer Conferences and Learning

Posted on by
1

In the past two years, “Peer conference” and “exhibition conference” are terms I’ve come to use a lot. A peer conference is a get together among practitioners of a particular discipline for the purpose of learning to practice better. It’s a round table, sometimes literally. An exhibition conference is what most people call just conference. There is often not much conferring going on at an exhibition conference. The point of most such “conferences” is for one class of people to put on a show for another class of people.

At an exhibition conference, it is considered impolite for someone to stand up and criticize the ideas of a speaker. It would be as if someone tried to talk back to the screen at a movie theatre. For many people attending exhibition conferences, there is a presumption that the speakers are experts. Thou Shalt Not Doubt Experts.

At a peer conference, everyone is a speaker, and it is expected that we will criticize each others’ ideas. We’re after deep learning, and it’s hard to get that without getting behind the Powerpoint glitz.

In my community, we’ve held sixty or seventy of these meetings since the first one in 1997 (the Los Altos Workshop on Software Testing, organized by Cem Kaner and Brian Lawrence). It’s become a vital mechanism for building relationships and developing our craft. It’s getting a little crazy, actually. In the last three weeks I’ve been to three peer conferences: The Workshop on Combinatorial Testing (WOCT), The Exploratory Testing Research Summit (ExTRS), and the Workshop on Training Software Testers (WTST).

Peer conferences create an environment for cross-organizational learning.

Recently, I’ve noticed a shift in the culture of the meetings. I’ve started using them as opportunities to challenge my colleagues to “testing duels” of various forms. Some of my colleagues have followed suit, so that when we meet, I can expect to be barraged by brainteasers or exhortations to test this or that with one or more hands tied behind n-squared backs. As a result, I get terribly sleep-deprived. It’s wonderful.

This is important if we want to build a better craft. For instance, I don’t think that the Agile community understands testing, yet. Testing is a skill of the mind, not a software tool that glows red or green with news of the trivial. I think the community of genuine software testers (those who devote themselves to the study and practice of dispelling illusions about software products) has a better hope of helping the Agile programmers understand what we do if we pose testing problems to each other. We should show those guys that not just anyone who falls off the backend of a compiler can expect to test with the big dogs.

I dream of a testing craft where bright young testers study hard to distinguish themselves at the next local testing tournament, the state championship, or the nationals.