Ever since I got into the testing field, almost 20 years ago, it’s been a truism that military software development is moribund. It’s not that they love process, it’s that they love bad process. Documentation? Bad documentation. Who can look upon 2167A or Mil-Std-499 without dismay? I’ll tell you who: paper mills and people paid to arrange the ink on all that paper. It’s just a scam and a travesty.
I was asked, in 1998, to analyze two military software software test plans, each for a different major weapons system. I told them up front that I was neither interested nor qualified to assess the test plans against DoD standards, such as Mil-Std-499. I was told, no problem, assess them against best commercial practice. Interpreting “best commercial practice” as what I would generally recommend doing for a life-critical or otherwise high stakes project in the commercial sector, I created a model for analyzing test plans (now published on my website and also in the appendices of Lessons Learned in Software Testing). I then applied the model to the test plans I was given. What was immediately apparent is that the military test documentation had very little information density. It looked like the 75-page documents had been automatically generated by a set of macros operating on a short bullet list.
I made a bunch of suggestions, including showing how the same information could productively be packaged in 5 pages or so. That way, at least we could wage war on something other than trees. They replied, after two months of silence, that my analysis was not useful to them. Why? Because my ideas about test plan documentation were not consistent with Mil-Std-499. That was the only feedback I received about that work. Way to squander taxpayer money, guys! Hoowaa!
A New Hope
The defense department may be waking up to the problem, at long last. See the NDIA Top Issues Report. Notice that two issues I like to harp about are in the top five challenges: skills and testing. Notice that the military is now officially concerned about wasteful documentation and low-skilled workers.
Maybe not coincidentally, I recently taught my class at Eglin Air Force base, with F-15s thundering regularly overhead. I was surprised that they would invite me to teach there. I was a bit more surprised that they were quite receptive to the concept of skilled and agile software testing, wherein our documentation is produce only because and only to the extent that it actually serves a useful purpose.
James,
It was with great interest I read your experiences of the military plannng process. It seems to me that whenever you get involved with large governent based institutions, (such as the NHS – National Health Service in the UK) that you are often faced with reams of beurocratic process and adherence to convoluted, overly complex and out of date procedures no matter how insane. (I wonder when “Mil-std-499” was written.) This in some sense counterpoints my experiences with most commercial businesses who seem to have problems enforcing an adhereing to good practice and process. Maybe its because these processes have been given to them rather than developed and matured from within.
What do you think, any ideas on why we seem have such extremes of bad practice.
Perhaps the process of change for government institutions is just much slower than a commercially driven business. Perhaps the commercail world has been driven to run before it can walk ,where as the other doesnt want to walk unless it has too.
Leading on from that, does rigoursly applied process narrow the mind and just give people an excuse for not thinking outside the box.
Mike Doel
[James’ Reply: Any sufficiently large institution, like a sufficiently large landform, creates its own weather. People working in very large organizations, especially if there is little turnover, tend to see their own experiences as defining the very limits of the world. Add money and ambition to this equation, and process work becomes something akin to a medieval landscape. Fiefdoms are established. Feudal lords reign. Learning, innovation, and high achievement take a back seat to bombast and ritual.
To counteract this effect takes a strong personality– someone like Kelly Johnson, who created Lockheed’s Skunkworks in order to escape the stultifying organization that would otherwise have prevented his crew from creating such masterpieces as the U2, the SR-71, and the stealth fighter.]
The problem seems to be growing, and not correcting, at least insofar as the civilian government goes. I’m not sure who started it, but the FAA gets the blame for the system that was going to redefine civil air traffic control in the US, and instead redefined the actual worth of the CMM as per James’ article. Then it spread to the Department of Justice, who used standards and paperwork to create the FBI’s new case file management system. It’s quite rare to do that, to be able to produce a system which, years later, contains absolutely nothing of any use at all. It has spread to the Department of Energy, and should preclude any useful automation happening there. But I’m sure it will all meet the latest government standards as it fails, runs over budget and over schedule.
The department of Defense also seems to have a study (I am trying to get a link to that report) that concludes that the waterfall method in as a SW development methdology had 0 (zero!) efficiency for their projects.
I might be making too many assumptions now, but I always thought that the Capability Maturity Model (CMM) was originally created by/for the Department of Defense. It seems that they never used the model to come to the conclusions they are making now.
On a seperate note, the FDA seems to start asking some questions. During one of the conferences with FDA participation, attended by my peers, the FDA people indicated that they wonder why medical devices have so many problems.
My answer to the FDA would be that their own guidance would be a direct cause of the issue the are investigating. As far as I know the FDA guidance does not address skill nor training. I believe that both elements should be a key for any guidance document.
During that same conference FDA also revealed the issue about document approvals
There seems to be a systemic problem of too many reviewers and approvers on documentation, and that too many lessens the constructive criticism from the review process. An informal study by an FDA inspector inferred that the quality of a document was inversely proportional to the number of signatures when that number was greater then three.
[James’ Reply: The SEI, which created the CMM, was funded by the Department of Defense for years. I don’t know whether they still are. The great traction that the CMM received has nothing to do with its merit. Adoption of it is limited almost exclusively to companies who gain a specific marketing benefit from it, or who seek preferred treatment on military contracts, which lamely require CMM “maturity”.
I have seen people who seem to have fire in their eyes about the CMM despite no obvious economic benefit to themselves due to CMM PR, but they are few and far between.]