Kevin asks: What is the best or industry standard for how many times a test case should be run?
There are questions that should not be answered. For instance, “What size unicorn do you wear?” or “How many cars should I own?” Sure, I could answer them, but the answers are worthless. My answers are A) I don’t wear unicorns and B) 2. In these cases, the more helpful reply is to question the question. For the first question, perhaps you said “uniform” and I misheard you. For the second question, perhaps you own a railroad and you were talking about train cars of different kinds, whereas I assumed you’re a small family and you were asking about automobiles.
I can tell you this for sure: No one I respect in the testing field will give you a direct answer to the general question of how many times a test should be run (except maybe as a joke).
Imagine if the answer was 100,000. Would you believe it? What if the answer was 7? Wouldn’t you wonder what was wrong with 6? I can imagine 7 being the right answer, but only for a very specific hypothetical case, not as any sort of general principle.
The first potentially useful answer I have is to tell you that this question would not even occur to you if you knew how to test, therefore, what you really need to do is start learning how to test. I mean if someone was re-wiring your house, and during that process he asked you what “voltage” is, wouldn’t you get someone else to wire your house? Like electrical work, plumbing, computer programming, or welding, good testing is a skilled activity.
I rarely give that answer, though, because I worry I will just leave people feeling discouraged.
The closest thing to a direct answer I can give you is this:
There exist no testing industry standards that are universally binding or even, in my opinion, more than negligibly helpful. Yes, there are documents that purport to be standards. If you are bound by them then you already know that. You aren’t subject to standards unless one has been imposed upon you by a regulating authority or by contract. Therefore, considering that testing costs money and time, I suggest that you don’t run any tests unless there is a reason to do so. In general, don’t do the same work a second time if you have already done it once. Certainly, if your clients would benefit from you running a test again, go for it. Otherwise, you are just indulging in compulsive/obsessive behavior, and you need help of a different kind than I offer.
A problem with this answer is that it begs the question of how you know when to run a test again. Fortunately, I wrote an essay on possible reasons to repeat tests. I can think of ten good reasons that you may want to repeat any given test (along with one big reason not to).
That’s a pretty good answer, but I think I can offer a little more:
Your job is probably to discover if there are terrible as-yet-unknown problems in your very complex product that you have little time to test. To do that job really well requires that you design and perform many tests, more tests than you probably have time to run. Therefore, when you run a test a second time, you are spending precious time and resources (even if it’s automated, though possibly less so) on something other than running a test you have not yet run that may find one of those big bugs you haven’t yet found. Get it?
So, how about having a small set of very basic tests that touch upon a lot features of the product. You may even want to automate these. It should take ten minutes to run these tests, ideally. Perhaps as long as an hour. Repeat those for every build. Their purpose is to quickly detect huge obvious things that may be wrong. Call that the smoke test suite. For everything else, make a test coverage outline that lists every significant element of the product and every significant element of data. Visit the items on that list and test each one according to its importance and potential for failure. Whenever any part of the product changes, try to figure out what could have been affected, and retest that area– but using different tests; perhaps variations on what you’ve already done.
By the way, the more you learn about testing, the less you will find advice like the preceding paragraph useful, because you will carry within you the ability to design your own test strategy that fits your specific purposes and contexts.
The job is to detect and to fix (in the sense of shepherding) software defects. It’s unbelievable how much nonsense you can cut through by just asking yourself, “What’s the most useful thing I could do next?” It’s not rerunning old or even variant tests that provide no new information. It’s not running a new test that won’t provide new information.
It’s not beetle-tracking in the name of “thoroughness”, without the use of the tester’s intelligence to guide the process effectively. If you’re not making progress toward the goal, then just exactly what are you doing?
The object is to find bugs. (Well, and to adminster the process of same.) If what you’re about to do isn’t the most effective thing you can think of towards that end, then you should just about certainly be doing something else.
If what you’re about to do is solely in satisfaction of some arbitrary metric, then you certainly should be doing something else, (That or you’re trying for an ISO-9000 qualification, which tells you what that standard’s worth.)
And if you’re repeating tests because you can’t think of anything else to do, it’s time for a long break, some meditation on your craft, and maybe a nice raw steak.
So look at it this way: if you’re doing the same old thing, you won’t find anything new to break, so you won’t make any coders cry, and what fun is that?
What about regression testing when we want to check if any old bug is reintroduced?
[James’ Reply: Okay, what about it?]
Do people really ask about how many times to run a test, without giving any other context? Asking about repeating a test on the same build of the software on the same configuration is different than asking about which of the different builds of the software a test should be run on or which configurations to run a test on.
[James’ Reply: That blog post was motivated by someone who did ask that question.]
Danny, I get asked unicorn questions all the time, and I see even more of them posted in public forums.
—Michael B.
Hear hear. A small “beautiful day” smoke set coupled with exploratory sessions sounds like an approach I would (and do) endorse.
Have you all considered manufacturing testing and reliability testing? I used to write diagnostic tests for the manufacturing and reliability groups. When I moved into other areas of code development, I brought along my test cycling habit. I found that cycling the same test, on the same software build, on the same hardware, could and did yield different results. Some bugs are intermittent and it would require blind luck to see the failure by running the test once. Is a 5% error rate acceptable? If not, how likely is it that you will see the error by running the test once? “How many times should I run a test?” is a perfectly valid question.
[James’ Reply: Please read the post again carefully, and follow the link that I provide in the post. You will see that I deal with this situation.
Of course, in some specific context, the question may be useful and may have a reasonable answer. My post deals with the question as a general, context-free qurey.]
Run the test case until it runs correctly. This may require a change to the test case, the requirement or the Hardware/Software.
[James’ Reply: You are kidding, right? Because that doesn’t address the question. It’s a pretend answer.]
Hi James I have a scenario:- In my years of experience I have seen ewhen we are building new products, whenever I run the regression test cases, I encounter new defects every time and the effort is huge in running the full regression cycle manually and based upon my limited knowledge on automation, what I understand is I cannot automate the regression bed because the properties of the object or the object itself will change, which can add to the maintenance of my automation suits using (Selenium/Renorex/UFT). In such scenarios I would like to understand what would be your approach
[James’ Reply: Without more information I can’t really answer this. My approach always involves a combination of tool use and testing skills, though.]