CNN has obtained a confidential status report on the development of healthcare.gov, which was sent from CGI Federal to Health and Human services, warning that their was not adequate time for testing before the October 1 launch date.

Status reports like this are very common for me. I author them myself to send out to clients, so that they are in the loop on what is happening with their project and money, so needless to say I have spent the morning tearing through it. A couple of items particularly stand out in the actual document (PDF file), this being the first:

(Click for larger view)

Websites have to run on servers. When writing websites, notably dynamic sites like healthcare.gov, that means you are writing software that runs in a specific server environment (language, language version, database, web server software, operating system software and versions, file permissions, etc.). It is a big difference from buying a program at the store and coming home and installing it on your PC. When developing software errors will and due occur. Sometimes you see them on the development machine you are working on, other times they aren’t visible until they are placed on a testing server. Luckily servers make it easy to find those problems by providing a log of all errors, but those logs don’t do any good unless you have a way to read them. In this case CGI didn’t.

Now before I go further, this does not exonerate CGI from the previous problems I have highlighted (and remember, this document was authored by CGI), but it does show they weren’t the only ones at fault here.

With that out of the way, back to this problem. CGI’s developers had no access to the server to even check needed items like logs, meaning they had no way to easily detect problems, short of spending hours or days of writing additional code to track down the problems. Even more troubling is that this is an issue that can be resolved in about 5 minutes, yet remained open for over two months, severely hindering development. Kathleen Sebelius really needs to be asked about this issue, specifically if the project lead at HHS was aware of it and what they were doing to rectify the problem between the various contractors.

Then we get to another issue. Sebelius has said numerous times now that the actual HUB has been working as designed. Well that may be the case since the rollout, but this issue here shows that wasn’t always the case:

(Click for larger view)

So with just a little over a month to go, at a time when testing should be the primary focus, that couldn’t even be done because of problems with the hub. When you consider that a key component of healthcare.gov is to communicate with this hub to verify user citizenship, income and other things, that’s a big problem. Add this to the first issue I mentioned. CGI is trying to test things and getting errors. They can’t check the error log, so is this a problem with their code, communications with the hub, or is it the actual hub overall? All those can be answered in under ten seconds with log access, but CGI was still waiting on that. 

Now that we know all this and with Kathleen Sebelius facing questioning on Capital Hill today, I believe we now have a couple of key questions that need answered:

• What, if anything, did HHS do to try and remedy the access problem to the servers between contractors?
  In a large project, such as this, problems between contractors are very common. It’s to say that the server access issue remained opened because of some feud between contractors. It could be as simple as the issue getting passed to some lower level employee and forgotten about. But this is why we have project leads, to not only organize everything, but also act as an intermediary between all parties. In this project that person was an employee of Sebelius and someone I’m sure she talked to daily, so why did this person apparently drop the ball on this issue?

• Did she inform President Obama of these serious problems, or did she push the warnings aside?
  I believe this question is key to Sebelius’ future. Delaying the actual launch of the website would have had serious political implications for the administration, but we are seeing those with the failures now, and in a time of uncertainty, we face a very real possibility that these implications could be far worse than a delay. Having said that, did Sebelius even warn the President of these problems so that he could have decided if we should delay or push forward? If she kept this information from the President, then Obama needs to call her in the Oval Office right now and ask for her resignation. The political fallout of that will be much less severe than continuing on a road of uncertainty with all the glitches still plaguing obamacare.

With a better understanding of what has taken place and what warning signs were there, we do know one thing for sure – the problems we face today are caused by a few factors:

• We have a bureaucratic procurement system that results in breakdown between contractors, which is something our government has had for decades. 
• There was a failure in the government level, at the project lead, with remedying problems that existed between contractors to keep things on schedule.
• The politics involved in healthcare.gov, especially the Republican effort to sabotage it, played a key role and was most likely a key factor in pushing forward with an un/under tested system.

Like I said at the beginning; status reports like this are something I’m intimately familiar with. These kind of reports are also seldom read by the needed people and only glazed over. I’ve gone through the very server issue that CGI mentions as recently as a few months ago. It took a couple of voicemails, numerous emails and finally a stark warning email with the subject “All work on your project is now suspended for violation of contract” before someone could be bothered to spend five minutes and fix the problem. There really is no excuse for this, except for laziness or just not caring enough, so the failure point in terms of the healthcare.gov project, the federal project lead in this case, must be held accountable.