The failure of Healthcare.gov is probably not because of sinister people. It’s probably because of the innocents who run the project. These well-intentioned people are truly as naive as little children. And they must be stopped.
They are, of course, normal intelligent adults. I’m sure they got good grades in school– if you believe in that sort of thing– and they can feed and clothe themselves. They certainly look normal, even stately and wise. It’s just that they are profoundly ignorant about technology projects while being completely oblivious to and complacent about that ignorance. That is the biggest proximal cause of this debacle. It’s called the Dunning-Kruger syndrome (which you can either look up or confidently assure yourself that you don’t need to know about): incompetence of a kind that makes you unable to assess your own lack of competence.
Who am I talking about? I’m talking to some extent about everyone above the first level of management on that project, but mostly I’m talking about anyone who was in the management chain for that project but who has never coded or tested before in their working lives. The non-technical people who created the conditions that the technical people had to work under.
I also blame the technical people in a different way. I’ll get to that, below.
How do I come to this conclusion? Well, take a look at the major possibilities:
Maybe it didn’t fail. Maybe this is normal for projects to have a few glitches? Oh my, no. Project failures are not often clear cut. But among failures, this one is cut as clearly as the Hope Diamond. This is not a near miss. This is the equivalent of sending Hans away to sell the family cow and he comes back with magic beans. It’s the equivalent of going to buy a car and coming back with a shopping cart that has a cardboard sign on which someone has written “CAR” in magic marker. It’s a swing and a miss when the batter was not even holding a bat. It’s so bad, I hope criminal charges are being considered. Make no mistake, the people who ran this project scammed the US government.
Did it fail because it’s too hard a project to do? It’s a difficult project, for sure. It may have been too hard to do under the circumstances prescribed. If so, then we should have heard that message a year ago. Loudly and publicly. We didn’t hear that? Why? Could it have been that the technical people kept their thoughts and feelings carefully shrouded? That’s not what’s being reported. It’s come out that technical people were complaining to management. Management must have quashed those complaints.
Did politics prevent the project from succeeding? No doubt that created a terrible environment in which to produce the system. So what? If it’s too hard, just laugh and say “hey this is ridiculous, we can’t commit to creating this system” UNLESS, of course, you are hoping to hide the problem forever, like a child who has wet the bed and dumps the sheets out the back window. I suppose it’s possible that Republican operatives secretly conspired to make the project fail. If so, I hope that comes out. Doesn’t matter, though. Management could still have seen it coming, unless the whole development team was in on the fix.
Were the technical people incompetent? Probably. It’s likely that many of the programmers were little better than novices, from what I can tell by looking at the bug reports coming through. It was a Children’s Crusade, I guess. But again, so what? The purpose of management, at each of the contracting agencies and above them, is to assess and assure the general competence and performance of the people working on the job. That comes first. I’m sure there were good people mixed in there, somewhere. I have harsh feelings for them, however. I would say to them: Why didn’t you go public? Why didn’t you resign? You like money that much? Your integrity matters that little to you?
Management created the conditions whereby this project was “delivered” in a non-working state. Not like the Dreamliner. The 787 had some serious glitches, and Boeing needs to shape that up. What I’m talking about is boarding an aircraft for a long trip only to be told by the captain “Well, folks it looks like we will be stuck here at the gate for a little while. Maintenance needs to install our wings and engines. I don’t know much about aircraft building, but I promise we will be flying by November 30th. Have some pretzels while you wait.”
Management must bear the prime responsibility for this. I’m not sure that Obama himself is to blame. Everyone under him though? Absolutely.
What About Testing?
Little testing happened on the site. The testing that happened seems to have confirmed what everyone knew. Now this article has come out, about what’s happening behind the scenes. I sure hope they have excellent Rapid Testers working on that, because there is no time for TDD or much of any unit testing and certainly no time to write bloated nonsensical “test case specs” that usually infect government efforts like so much botfly larvae.
Notice the bit at the end?
“It’s a lot of work but people are committed to it. I haven’t heard anyone say it’s not a doable job,” the source said of the November 30th deadline to fix the online portal to purchase insurance on the federal exchange.
Exactly. That’s exactly the problem, Mr. Source. This is what I mean by the tyranny of the innocents. If no one is telling you that the November 30th deadline is not doable, and you think that’s a good sign, then you are an innocent. If you are managing to that expectation then you are a tyrant. It’s probably not doable. I already know that this can’t possibly leave enough time for reasonable testing of the system. Even if it is doable, only a completely dysfunctional project has no one on it speaking openly about whether it is doable.
What Can Be Done?
Politics will ruin everything. I have no institutional solution for this kind of problem. “Best practices” won’t help. Oversight committees won’t help. I can only say that each of us can and should foster a culture of personal ethical behavior. I was on a government project, briefly, years ago. I concluded it was an outlandish waste of taxpayer money and I resigned. I wanted the money. But I resigned anyway. It wasn’t easy. I had car payments and house payments to make. Integrity can be hard. Integrity can be lonely. I don’t always live up to my highest ideals for my own behavior, and when that happens I feel shame. The shame I feel spurs me to be better. That’s all I’m hoping for, really. I hope the people who knew better on this project feel shame. I hope they listen to that shame and go on to be better people.
I do have advice for the innocents. I’ll speak directly to you, Kathy Sebelius, since you are the most public example of who I am talking about…
Hi Kathy,
You’re not a technology person. You shouldn’t have to be. But you need people working for you who are, because technology is opaque. It may surprise you to know that unlike building bridges and monuments, the status of software can be effectively hidden from anyone more than one level above (or sideways from) the programmer or tester who is actually working on that particular piece of it. It’s like managing a gold mine without being able to go down into the mine yourself.
This means you are in a weak position, as an executive. You can pound the table and threaten to fire people, sure. It won’t help. The way in which an executive can use direct power will only make a late software project even later. Every use of direct power weakens your influence. Use indirect power, instead. Imagine that you are taming wild birds. I used to do that as a kid in Vermont. It requires quietness and patience. The first part is to stand for an hour holding birdseed in your hand. Stand quietly and eventually they are landing in your hand.
To have managed this project well, you needed to have created an environment where people could speak without fear. You needed to work with your direct reports to make sure they weren’t filtering out too much of the bad news. You needed to visit the project on a regular basis, and talk to the lowest level people. Then you needed to forgive their managers for not telling you all the bad news. It’s a maddeningly slow process. If you notice, the Pope is currently doing something very similar. Hey, I’m an atheist and yet I find myself listening to that guy. He’s a master of indirect leadership.
You did have the direct power to set expectations. I’m sure you realize you could have done a much better job of that, but perhaps you felt fear, yourself. As your employer (a taxpaying citizen), I bear a little of that responsibility. The country is getting the Healthcare.gov site that it deserves, in a sense.
If you are going to continue in public service, please do yourself and all of us a favor and take a class on software project management. Attend a few lectures. Get smart about what kinds of dodges and syndromes contractors use.
Don’t be an innocent, marching to the slaughter, while millions of dollars line the pockets of the people who run CGI and all those other parasite companies.
— Sincerely, James
My Political Agenda
I have $200,000 of unpaid medical bills due to the crazy jacked up prices and terrible insurance situation for individual citizens in the United States. I am definitely a supporter of the concept of health care reform, even the flawed Obamacare system, if that’s the best we can do for now.
I was pleased to see the failure of the Healthcare.gov website, at first. A little failure helps me make my arguments about how hard it is to do technology well; how getting it right means striving to better ourselves, and no formula or textual incantation will do that for us.
This is too much failure! I want it to stop now. Still, I’m an adult, a software project expert and not in any way an innocent. I know it’s not going to be resolved soon. No Virginia, there won’t be a Healthcare.gov website this Christmas.
Addendum:
From cnn.com:
Summers wrote a memo to the President in 2010 suggesting that HealthCare.gov was not something the government could handle and he needed to bring in experts.
While Summers would not provide details about internal discussions, he said Tuesday, “You need experts. You need to trust but you need to verify. You can’t go rushing the schedule when you get behind or you end up making more errors.”
Damn straight. If this is true then I’m sure glad someone around Obama had basic wisdom. I guess nobody listened to him.
James, I was troubled by all these vulture consultants immediately fanning across the gullible media (Washington Post, New Yorker, Politico, etc) suggesting without a hint of knowledge of the situation that the Waterfall boogeyman was to blame and Agile could have cured all the ills.
Michael Danconta at GCN seemed to have the scoop on debunking the vultures.
[James’ Reply: I’m troubled by that, too. And I don’t want to be part of a facile piling on. Technology is hard and I respect that. But… we can now plainly see that this project was miles off. Miles and miles. The people in charge either knew a year ago that they couldn’t be ready in time, or they just didn’t have the ability to understand these things, nor to hire people or listen to people who might have had that ability. Villains, or more likely, innocents.
It certainly isn’t about Agile or Waterfall. The problem is so much deeper than that. I remember working for a consulting firm outside of D.C. years ago, that bid on a project I told them to stay away from. It was actually my first day at work and I saw the RFP and told my bosses “This project cannot succeed. Do not bid on it.” Well, they did anyway. I bailed out of it after six months, and the project was canceled 18 months after that. The problem there was rapacious greed, coupled with the cynical belief that blame can always be shifted to someone else.
This is an all too common problem in the world of big-time consulting. As long as no one ever goes to jail, it will continue to be a problem.]
About ten years ago, I was test manager on a CRM platform built on contract for CMS. I blogged about my experience the other day. It was a command-and-control environment, both through CMS and at the company I worked for, and I find that this is a terrible environment in which to deliver software because it encourages and rewards behaviors that hide the actual state of the project. Moreover, I found the government to be afraid to death of being screwed over by its contractors, and so we had mountains upon mountains of compliance- and audit-trail-related stuff to do, just to prove that we weren’t screwing them over. It made testing take multiple times longer than it needed to, and most of the things my testers were therefore forced to do did not materially help us find defects in the software.
In the end, we delivered fairly successfully. I say fairly because v. 1.0 had some distinct quality challenges that we fixed over the next several patch releases. But it was an expensive and ponderously slow way to deliver software.
[James’ Reply: I noticed that about government work, too. Everyone IS trying to screw the government, and government IS trying hard not to be screwed, and that makes projects that should be afternoon strolls a process more like inching through a minefield.]
Hmmm, while I appreciate your delineation between innocents vs. non-innocents, me thinks it may be more appropriate to draw a focus on the guilty.
Generally, my perspective is that no actual “innocent children” were at work on this project, a foregone conclusion implied by the child labor regulation in this country.
You may be right to be kind to those adult professionals with decades of experience as being innocent, but I’m not that kind.
Naive? Sure.
Ignorant? Most certainly.
Culture of squelch and silencing? Believable.
But, innocent? Not in my humble opinion.
Great post – thanks James!
[James’ Reply: In an important sense, I agree that they are not innocent. I’m angry with them and they deserve our anger. But at the same time, at a certain level of ignorance, it becomes kind of meaningless to treat people as consenting adults. You’re just kicking a puppy, and the puppy doesn’t know what it did wrong.
My contempt for the willfully ignorant is well documented, though. I sure hope criminal charges are forthcoming, just because that will help wake up future generations of such managers.]
Hi James,
A thought provoking post as ever. The thought I’m having right now is: How does this square with your advice for us testers to get out of the quality business (and ship/don’t ship decisions)?
[James’ Reply: This isn’t about testing at all. They knew the thing didn’t work.]
What I’m trying to square, is my responsibility to provide quality information to my decision making stakeholders, with knowing how hard to push back when they don’t act on that information (or more accurately, appear not to). It’s not always as clear cut as this example seems to be.
[James’ Reply: I think if you push back at all you are not being a tester (and that could be okay). A tester’s job is to present the information. Once that information is received, it’s not your problem (at least not your testing problem, although you may feel that as a citizen or employee of a company you have other reasons to act).]
Let me know what you think.
Vern
James, your writing here is some of the best I have seen from you in ages. I very much appreciated your even handed handling of the material. It really reminded me of The Gervais Principle (http://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/ ). In a similar set of delineations, the author talks of the Sociopaths, the Clueless, and Losers. The clueless don’t know and don’t know they don’t know. The losers (economically speaking) choose to ignore work, not because they lack honor, but because they find value somewhere else in life. The sociopaths, for better or worse see themselves independent and unaccountable for anyone but themselves. In that imperfect lens, the communication breakdowns start to make sense.
[James’ Reply: I love reading Ribbonfarm. I almost never agree with the guy, but he is so thoughtful while being wrong! I am always learning something from him. The Gervais principle seems to me like a oversimplistic analysis that has a lot of insight in it. I particularly don’t feel that people who run companies are necessarily sociopaths, although that seems the only option. I run a company. I disagree that makes me a sociopath.]
Without getting into politics, the question in my mind is, do you ignore ‘unchangeable’ issues in hopes to change the system slowly from within (or get high enough to change the system yourself), abandon the system to go do something else or actively attempt to break the system and replace it with something else. I’m not sure your choice of leaving the system is any better than staying and fighting for change. Actively breaking the system, whistle blowing, clearly is not the most healthy activity as Snowden and others have demonstrated. I’m not sure there is an answer, but I think the questions are worth contemplating.
[James’ Reply: I can’t say that any particular way is better, universally. I can market my values, though, and hope others will join me. I can also reason from those values.
I have a particular personal attribute that makes the decision easier for me than most people: my temper. I stay away from situations that are bad and that I can’t change, because I don’t want to get angry. It’s like a severe allergy. I have never been arrested and don’t plan to be. I like to support people who stay in bad situations, though. Although I can’t do it, I appreciate people who can stay in and try to make things better.
However, some things cannot be saved. I don’t think it’s possible to change the ISTQB organization. I think it was conceived in depravity and is depraved by nature. And perhaps the Healthcare.gov project was like that. I don’t know.]
According to the linked cnn article (http://politicalticker.blogs.cnn.com/2013/11/12/behind-the-scenes-fixing-the-obamacare-website/), they are short on printers. Maybe they are slowed by not being able to print their test cases. They are probably taking longer on their code reviews without good printouts. Possibly they aren’t getting their sign-off documents don’t quickly enough. I guess they aren’t able to print their certifications out. Of course the gantt charts, which are difficult to print in the best of circumstances, are holding up the project.
Oh, if they could only have more inkjets and laserjets, then it would all be OK.
In my professional life I’ve left 3 projects prematurely. Two in govt and one in big industry. All had some easily spottable characteristics:
1) You were ordered to silence.
a) You have to sign an NDA
b) You get told by management to shut up or else
c) You get told to bury information/destroy information
d) You get a mantra to repeat like “All is going swimmingly”
2) Logic doesn’t seem to be a valid answer
a) You present clear and logical numbers proving that things can’t get done as planned. You get TOLD that it WILL work
b) You get asked how many people you need to make “IT” happen
c) You get screamed at or called names (sometimes only behind your back)
3) You are not allowed to talk to your customer or get into trouble for doing so
If any of those apply think about what you are doing and what the future holds for you. And don’t be fooled by smiles and presents.
NZ just had a similar failure with Novopay and all the hallmarks are there too. What I’ve now been on about for the last 3 years is exactly what James points out above. The mortgage problem. Not everyone has the guts it takes to put your family and liveliehood alas your future on the line for your job. It really isn’t for everyone and I usually do understand if people don’t BUT it does make them co conspirators they do need to accept that and all that that entails (you are not innocent. You’re the guy that get’s offered the deal by police, if things go south 😉 ).
I’ve been thinking and talking about how we mitigate the mortgage problem. It is actually very similar to what has been going on over the past years with Whistle-blowers. There needs to be a test/IT whistle-blower clause. Some way we can actually start to deal with catastrophic failure openly, to the advancement of the project and IT.
Currently we all would still face possible legal consequences from doing the right thing.
Dear Mr. Bach,
I know (a little bit) about you, Mr. James Bach, and I have found myself disagreeing with you some over the years. So it was with a wee bit of trepidation that I opened your blog post. As I read it, however, I found *very* little to disagree with, and found myself nodding my head silently.
Yes, this is another one which we can chalk up to “management failure”, but it is not the type of “management failure” that most industry consultants prattle on about.
It is the failure of senior professionals to properly “manage their managers”, their failure to properly mentor their juniors, and quite frankly, their failure to act as professionals. Sometimes, as you imply, that necessitates resigning. More often, as I have experienced personally, it necessitates an attitude of “I’ll be darned if I’m gonna quit, I’m gonna MAKE them fire me!”, as one adopts an attitude of “doing the right thing”, but without being explicitly and overtly insubordinate.
Years ago, a team leader who had been promoted to his level of incompetence, a manager, a la the Peter Principle, called me at my cube and asked me to do still another pointless, fruitless thing. Now I’m one who at least tries to be charitable, and I thought real hard trying to get inside his brain and make sense from any “bigger picture” point of view, and I just could not come up with something. Now maybe I should have just directly challenged this guy with a “this doesn’t make sense”, but I had already conferred with another senior peer on the project who independently was coming to the same conclusions as I. Then I had a flash of insight – this manager was JUST ANOTHER VOICE ON THE PHONE. Gee, I had all kinds of end users in several other departments, and I was keeping them happy and off his back, and all kinds of John Q. Public’s who were benefiting and gainfully advantaged by what I was doing. So I adopted an attitude that I would listen to my manager, but no more than anyone else, and it was my job as a professional to balance all these interests. OK, eventually he let me go, but that was three (or more) years later, and in the meantime I was able to help a whole lot of people.
A few years after that incident, someone asked me, “What is it to be an IT professional?” And reflecting upon my experiences, I instantly knew the answer: “An IT professional is one who seeks, and attains, a harmonious balance among his self-interest, the interests of his clients, and the interests of his peers. If you can do that, then no matter what else, you have had a successful project.”
Now I make no claim that if more people had adopted this attitude, then the ACA project could have been “saved.” But I am very confident that if more people had adopted that attitude, then the assets developed thus far, including the “development culture” that created them, would be (even) more salvageable than they are now.
Kenneth Shafer
American Flier, Inc.
Hi James,
Still, to write my comments on your analysis of the Healthgov. debacle! hopefully, will get around it !
but one question: about your introduction of Dumming-Kruger Effect, don’t you think that if “work ” to become as a right, like right to basic education or access to basic health, in our Industrial socities, then we might be able to see diminution of at least one aspect of this “syndrome”? – i am talking about the over-exaggeration aspect of this syndrome. Because the other aspect i.e. expert underestimating his or her abilities might not a bad thing to have in our World. It makes us more human.
take care,
[James’ Reply: I’m not sure what you are saying. I think Dunning-Kruger is about being unable to comprehend your own lack of judgment. I don’t see how this will diminish unless there is a much greater social pressure to learn and more general social awareness that there is anything important to learn.]