A lot of people I teach seem to be under pressure to create more documents to describe their test process. But documenting is not testing. It is one of the chief distractions to testing.
“James Bach hates documentation!”, some people will say. No, I don’t. I hate waste. I hate boring clerical work that unnecessarily interrupts productive and interesting intellectual work. You should hate it too.
I’m a writer for cryin’ out loud. I like documentation– when it is the solution to a problem that matters, and not merely a management obsession. But if you’re trying to move to concise test documents (outlines, matrices, one-page reference sheets, and other minimalist formats) you may need help persuading management and co-workers.
Here are some ideas:
- Show management how much less testing we are able to do because we are spending so much time with documents.
- Show management how certain kinds of testing isn’t done at all just because it is hard to document (exploratory testing and complex scenario tests often fall in this category). This is perhaps the most chilling effect of over-documentation, especially in the realm of medical devices. I keep seeing medical device test doc that is simplistic, in all its obesity, to the point of near worthlessness.
- Examine closely what testers are doing and show that they aren’t even following the documentation (often they aren’t, in my experience as a consultant who audits test processes).
- Demonstrate the power of exploratory testing (a less heavily documented approach). One day of ET is often sufficient to find what would take a week to find when following detailed documented test procedures.
- Demonstrate the value of concise test documentation (matrices, outlines).
- Consider documenting at the level of test activities rather than test cases.
- Consider automatic documentation (via log files produced by the product under test or via an external logging tool such as Spector).
- Ask the question: what exactly are we getting from our documentation? Don’t accept any theoretical answers. For example, one typical answer is that documentation protects the company from the ill effects of staff turnover. But does it? Probably not, in my experience. That’s a theory based on ignorance about how people learn. In real life, new testers learn what they need to know by playing with the product itself, and talking to the people around them. In my experience, testers come up to speed in a few days at most. And in my experience, test documentation is often of such poor quality that it’s better ignored than followed. You have to go by your own experience, of course. I’m just suggesting that you ask the questions and make your own observations.
Here are a few books about documentation that might help you make your case:
Heavy documentation is often a consequence of managers and testers who just aren’t thinking through the reasons why they do things. They hear that documentation is good, but they don’t stop to consider the cost of documentation, or watch how documentation is actually used (or more often, ignored).
From my perspective , I see that documentation is very much required. Yes Bad documentation is Bad but good documentation is also Good.It is just not to secure a turn over , but shows necessary traceability , displays more often the coverage of tests, to understand the weightage of tests written for the various modules. Documentation should be a part of the testing and if we find documents is taking a lot of time, then its upto management to plan their test schedule including the time required for the documentation.
Exploratory testing should be off the shelf outside the documents and interesting test scenarios that has identifies issues should again be documented. Thats my cents of experience.
Agree with James on most of his points ,:”Demonstrate the value of concise test documentation (matrices, outlines). ” and ” Bad documentation is not good ” and yes smart documentation is required.
For me personally, there is a tension between the (perceived) need to document what I’m doing and actually getting on with the testing work itself. With generally short periods of time in which to “do testing” anything that takes you away from interacting with the software under test, especially admin, is assumed to be negative. The problem is do the need to keep track of what you’ve done in a systematic way without it interfering or steering your progress too much.
However this pales into insignificance compared to having to come up with a test plan/strategy based on documentation alone, before you even get to see the product. Depending on the personality trait of the tester this can either be a useful exercise or some form of mental torture. I have yet to experience for myself or witness in others the former.
The hard thing to reconcile is that formulation of a test plan/strategy makes sense from a control or management point of view but it does not represent how people think or behave in the real world. Yet if a company has a policy for producing test documentaiton then what tends to happen is that rather poor and basic test plans are written because the process is seen to be just another box-ticking exercise. This is because it may have lot of theoretical value but this is rarely borne out in practice. Sometimes, if a detailed and accurate specification can be worked from (very rare!), the corresponding test plan can be pretty thorough. However I’ve yet to see anyone actually follow one that closely: it’s just a base for jumping off. But if that’s the case, why not just use the specification as a springboard and form a test plan as you go along? Ideally testers should inspire each other with their documentation but truth be told if people hardly follow their own plans, you can be pretty sure they almost never follow someone else’s mess either.
To sum up, I’m not saying don’t document; I’m saying don’t fret about documenting what you’re going to do before you do it. Even if you can come up with something decent, it will almost immediately be undone by the process of actually using the product under test.
[James’ Reply: Useful thoughts. Thanks.]