Government US

Healthcare.gov web site launch leaves lessons for health of your IT career

While a lot of people are furious at the mess that was the healthcare.gov web site rollout, IT pros should use it as a learning experience for their own IT careers.

IT pros following the troubled launch of healthcare.gov can only feel grateful that they were looking at it purely clinically, and as “outside observers”—especially if they were reading some the criticism about the web site in national publications. The President was even quoted in international presses as he acknowledged the web site’s problems, saying "There's no sugarcoating it. The web site has been too slow. People have been getting stuck during the application process. And I think it's fair to say that nobody is more frustrated by that than I am."

Nevertheless, if your career is IT, you look to learn from it.

The Washington Post published a detailed technical analysis of what went wrong. The conclusion was that there was a wide range of systems integration, web site performance, testing, usability, communications and “creeping enhancement” issues that were never adequately addressed. Collectively, these issues snowballed until the project spun out of control—but a call was made for the web site to “go live” anyway.

It made me reflect on an earlier moment in my career, when I was asked to assume the project manager position for a Wall Street online trading system project that was failing. The client had already sunk millions of dollars into the project. It had been told the project was “nearly ready,” but when I evaluated the project as a new manager taking over, I could see that the project wasn’t close to being ready.

I held a meeting with the client’s Executive Vice President and told him the situation. I can still remember him threatening to throw me and my lead managers (who were also with me) out the window! (His office was on the 46th floor.) Nevertheless, we all went back to our respective companies and presented the unvarnished facts—and we set the project back so it could be properly tested and prepared for launch. When it launched six months later, it worked—without a hitch.

This lesson has stayed with me throughout my IT career—and it should have been operative for healthcare.gov. In other words, never let a highly visible project with so much at stake fail! First, this magnitude of failure is a career risk for someone (and he’s usually working in IT). Second, failure allows project detractors to thrive.

There are other “IT lessons” that this project also delivers. They include:

Testing and quality assurance

We preach testing and QA for systems, but let’s face it: when project timelines get tight, stuff can get “thrown over the wall” just to meet deadline. I’ve received new software releases from major technology vendors that included modules that weren’t even compiled, let alone tested!

In the case of healthcare.gov, reports were already surfacing days and even weeks before the launch of the Website that it hadn’t been properly tested.

This testing should have occurred on several levels:

  • End user experience (EUE) testing to ensure that the Website was easy to use (it proved not to be);
  • Integration testing to ensure that 112 different systems and the efforts of 55 different vendors were all working together correctly (they weren’t); and
  • Stress testing that would have ensured that the system was sufficiently robust to handle more than the 14.6 million unique user visits that the system experienced in its first ten days of operation (it couldn’t).

Vendor management

There were 55 different contractors for this project. When congressional hearings commenced, no one asserted a “lead role,” and it’s still unclear who ultimately was responsible for the project.

Some placed blame on government procurement practices that didn’t emphasize the key success factors needed to make an IT project like this work. Whatever the issues, a federation of 55 different contractors with no one clearly leading is too many participants to manage in a project with such heavy integration requirements. It’s also unclear what SLAs (service level agreements) were defined in vendor contracts, but clearly some SLAs should have addressed work product integration with the overall system.

Enhancement control

At some point, project managers have to freeze enhancements so projects can be fully tested and released. Apparently, there were late calls for enhancements just weeks before healthcare.gov went live. This is where the project manager needs to make the unpopular call of pushing the project back when he can’t guarantee the quality of the results.

About

Mary E. Shacklett is president of Transworld Data, a technology research and market development firm. Prior to founding the company, Mary was Senior Vice President of Marketing and Technology at TCCU, Inc., a financial services firm; Vice President o...

16 comments
Cicuta2011
Cicuta2011

55 contractors is nothing really imagine how many contractors work on government defense and NASA projects – a good example is the Space Shuttle, Trident submarine, etc. We are talking about more than 55 contractors and at the end the system works as planned. So, what is the secret about being successful in projects of such magnitude? Management, development (R&D), and testing at the contractor and subcontractors levels and then “Integration” of all the subsystems and more testing in the lab and in real life and not launch the complete system till the bugs have been resolved. For such projects, there has to be what is called: “Milestones Management” which includes the main contractor and all sub-contractors in the project. Each milestone has to meet a dead line also.

What is clear to me is that in this case there was no contractor and subcontractor management at all and for sure “Milestones Management” was neglected as well as testing in the lab and real life. Everybody went about blindly and at the end failure is eminent.

It has to be noted also that IT is not a new science and for sure no human life is at stake contrary to defense and space projects; hence there is no risk at all…just negligence!

brf531
brf531

There is a related thing that occurs in any situation where you have multiple vendors, especially a lot of them.  We in AT&T Labs used to call it "Range Chickens."  Each behind-schedule part of the project hopes beyond hope that someone else will get nailed first and therefore let them off the hook.  Folks here are right, a strong project manager (who is a technical person, not a "manager") is the best solution.

lpazdern
lpazdern

Actually, the success rate for these large scale  IT projects is abysmally low. It's impressive the site functions as well as it does, and the problems are being addressed to remedy the issues. Sure it's easy to say that more testing should have been done, and that's what the contractors wish to blame it on, but testing alone wasn't the answer. The contractors involved should have clearly announced they needed additional time, but they were clearly up against the wall and fighting a losing battle from the start.

Al K
Al K

The lesson to learn is hire the salesman who convinced them to spend $600 million on the site and then gets a commission on all the $ spent to actually make it work.

jskopek
jskopek

LOL! Ha! I clicked on the link "technical analysis of what went wrong" and got "Page not found". I thought that was the Washington Post article title! Looks like Tech Republic has some link checking of it's own to conduct (unless you guys just wanted to make a funny...).

JohnMcGrew
JohnMcGrew

Good piece.  In short, a typical government cluster, with politically-driven and changing goals, lots of money thrown at it and nobody actually in charge.  Can't wait until these people are running everything.

darko.cepun
darko.cepun

Scope creep, why it happens and how to avoid it is one of the first lessons that PMI (Project Management Institute) teaches an IT professional. That’s the primary reason for failure for the most of the failed software projects – and this was a software project. So reasons for this to happen is pure lack of knowledge, I think.

Pinkwho
Pinkwho

I would say stress testing was the overall downfall, a user can get around a difficult to use site, and the integration issues could be handled at a maintenance window, but to have the site go down because of too many users? This is an infrastructure issue. Did they not put up multiple load balanced web servers? It seems with the correct infrastructure and stress testing, it would be an annoyance rather than a complete fiasco.

ChallengerTech
ChallengerTech

How did this happen? Zero project management, that's how! HHS was not managing their contractor(s) that were awarded the job. The contractors were not managing their project. Plain and simple.

cgilmore
cgilmore

Most people think the complexity is the problem in keeping projects on track.  In reality the problem is most practices in IT, condone lying to make deadlines.  Simply put PMO needs auditors to keep compliance in line with reality.  There is no enforcement mechanism to keep distortion down to a min. 

RMSx32767
RMSx32767

Perhaps AlGore should have been in charge; after all, he invented the internet. ;-P

BHPi-BA
BHPi-BA

Hi! You really "hit the nail on the head' with your overview of all that went wrong with the rollout of healthcare.gov.  Your analysis of this fiasco was spot-on.  Thank you for sharing your keen insight.

Navy MBA PhD
Navy MBA PhD

@lpazdern How low?  What is your empirical evidence?  I personally led a pm team to simultaneously deploy an inpatient EHR and electronic anesthesia record to more than 65,000 end users located on 4 continents in 6 countries and 19 regional hospitals.  I did it through sound project management, good communication, and a small amount of fortuitous timing.  


Healthcare.gov clearly had no leadership, no direction, and very little management -- directly leading to poor communication and an abysmal failure.

FTAdmin
FTAdmin

@jskopek  

The TR link is failing because of a dot (.) at the end of it. I.e.: "... graphic.html." as opposed to "... graphic.html" Removing that ending dot and hitting enter brings up the page.

ahadenfeldt
ahadenfeldt

@jskopek Same for me, but once I found it their Twitter-sharing tinylink seems to work consistently: http://wapo.st/19BQPKw

Can't say I'm surprised at some of the root cause issues--I've certainly learned the value of a truly effective project manager. There's a related article from 11/2 (longer & less technical) that dives into those upper-layer issues and is perhaps even more disturbing: "HealthCare.gov: How political fear was pitted against technical needs" http://wapo.st/1faZGF7


zwayne
zwayne

@Pinkwho  @Pinkwho Stress testing was not the overall downfall - guess you haven't really been following this fiasco. Too many users was the original reason given for the problems. But it didn't work even when very late testing was done with small amount of users. This site was - AND STILL IS - broken on just SO many levels.