To Repeat Tests or Not to Repeat

One of the serious social diseases of the testing craft is the obsession with repetition. Is that test repeatable? Is that test process repeatable? Have we repeated those tests? These questions are often asked in a tone of worry or accusation, sometimes accompanied by rhetorical quips about the importance of a disciplined process– without explanation of how discipline requires repetition.

(Before you go on, I urge you to carefully re-read the previous paragraph, and notice that I used the word obsession. I am not arguing with repeatability, as such. Just as one can argue against an addiction to food without being against eating, what I’m trying to do is wipe out obsession. Please help me.)

There is one really good reason not to repeat a test: the value of a new test is greater than the value of an old test (all other things being equal). It’s greater because a new test can find problems that have always been in the product, and not yet found, while an old test has a non-zero likelihood of revealing the same old thing it revealed the last time you performed it. New tests always provide new information. Old tests sometimes do.

This one powerful reason to run new tests is based on the idea that testing is a sampling process, and that running a single test, whatever the test, is to collect a tiny sample of behavior from a very large population of potential behaviors. More tests means a bigger sample. Re-running tests belabors the same sample, over and over.

Test repetition is often justified based on arguments that sound like blatant discrimination against the unborn test, as if manifested tests have some kind of special citizenship denied to mere potential tests. One reason for this bias may be a lack of appreciation for the vastness of testing possibilities. If you believe that your tests already comprise all the tests that matter, you won’t have much urgency about making new ones.

Another reason may be an innappropriate analogy to scientific experiments. We were all told in 5th grade science class about the importance of the controlled, repeatable experiment to the proper conduct of science. But what we weren’t told is that a huge amount of less controlled and less easily repeated exploratory work precedes the typical controlled experiment. Otherwise, an amazing amount of time would be wasted on well controlled, but uninteresting experiments. Science embraces exploratory as well as confirmatory research.

One thought experiment I find useful is to take the arguments for repetition to their logical extreme and suppose that we have just one and only one test for a complex product. We run that test again and again. The absurdity of that image helps me see reasons to run more tests. No complex product with a high quality standard can be considered well tested unless a wide variety of tests have been performed against it.

(You probably have noticed that it’s important to consider what I mean by “test” and “run that test again and again”. Depending on how you think of it, it may well be one test would be enough, but then it would have to be an extremely complex test or one that incorporates within itself an extreme amount of variation.)

The Product is a Minefield

In order to replace obsession with informed choice, we need a way to consider a situation and decide if repetition is warranted, and how much repetition. I have found that the analogy of a minefield helps me work through those considerations.

The minefield is an evocative analogy that expresses the sampling argument: if you want to avoid stepping on a mine, walk in the footsteps of the last successful person to traverse the minefield. Repetition avoids finding a mine by limiting new contact between your feet and the ground. By the same principle, variation will increase the possibility of finding a mine.

I like this analogy because it is a meaningful and valid argument that also has important flaws that help us argue in favor of repetition. The analogy helps us explore both sides of the issue.

In my classes, I make the minefield argument and then challenge students to find problems in it. Each problem is then essentially a reason why, in a certain context, repetition might be better than variation.

I won’t force you to go through that exercise. Although, before you click on the link, below, you may want to think it through for yourself.

I know of nine interestingly distinct reasons to repeat tests. How many can you think of?

Click this link when you are ready to see my list and how the argument applies to test-driven design: Ten Reasons to Repeat Tests