Comments on: Question: How Many Times Should You Run a Test?

By: What? Only run a test once?

What? Only run a test once? — Thu, 12 Jan 2012 12:09:10 +0000

Have you all considered manufacturing testing and reliability testing? I used to write diagnostic tests for the manufacturing and reliability groups. When I moved into other areas of code development, I brought along my test cycling habit. I found that cycling the same test, on the same software build, on the same hardware, could and did yield different results. Some bugs are intermittent and it would require blind luck to see the failure by running the test once. Is a 5% error rate acceptable? If not, how likely is it that you will see the error by running the test once? “How many times should I run a test?” is a perfectly valid question.

[James' Reply: Please read the post again carefully, and follow the link that I provide in the post. You will see that I deal with this situation.

Of course, in some specific context, the question may be useful and may have a reasonable answer. My post deals with the question as a general, context-free qurey.]

By: Petteri Lyytinen

Petteri Lyytinen — Wed, 16 Nov 2011 10:01:54 +0000

Hear hear. A small “beautiful day” smoke set coupled with exploratory sessions sounds like an approach I would (and do) endorse.

By: Michael Bolton

Michael Bolton — Wed, 17 Mar 2010 05:14:07 +0000

Danny, I get asked unicorn questions all the time, and I see even more of them posted in public forums.

—Michael B.

By: Danny Faught

Danny Faught — Fri, 17 Apr 2009 15:14:55 +0000

Do people really ask about how many times to run a test, without giving any other context? Asking about repeating a test on the same build of the software on the same configuration is different than asking about which of the different builds of the software a test should be run on or which configurations to run a test on.

[James' Reply: That blog post was motivated by someone who did ask that question.]

By: Pajton

Pajton — Wed, 05 Sep 2007 08:35:40 +0000

What about regression testing when we want to check if any old bug is reintroduced?

[James' Reply: Okay, what about it?]

By: insectivorous

insectivorous — Tue, 03 Oct 2006 16:48:27 +0000

The job is to detect and to fix (in the sense of shepherding) software defects. It’s unbelievable how much nonsense you can cut through by just asking yourself, “What’s the most useful thing I could do next?” It’s not rerunning old or even variant tests that provide no new information. It’s not running a new test that won’t provide new information.

It’s not beetle-tracking in the name of “thoroughness”, without the use of the tester’s intelligence to guide the process effectively. If you’re not making progress toward the goal, then just exactly what are you doing?

The object is to find bugs. (Well, and to adminster the process of same.) If what you’re about to do isn’t the most effective thing you can think of towards that end, then you should just about certainly be doing something else.

If what you’re about to do is solely in satisfaction of some arbitrary metric, then you certainly should be doing something else, (That or you’re trying for an ISO-9000 qualification, which tells you what that standard’s worth.)

And if you’re repeating tests because you can’t think of anything else to do, it’s time for a long break, some meditation on your craft, and maybe a nice raw steak.

So look at it this way: if you’re doing the same old thing, you won’t find anything new to break, so you won’t make any coders cry, and what fun is that?