Security

Worst IT fail ever? US agency spends millions in useless security

A branch of the U.S. Commerce Department recently trashed perfectly good computers and wasted millions on a bogus malware infection. How did this happen?

Hackers and malware are everywhere, waiting for you around every corner of the Internet. It's great to be paranoid as long as you know what you're doing, but someone with only basic knowledge of IT who browses through the constant security bulletins, security mailing lists, and even their own system logs could be overwhelmed quickly. This is apparently what happened when the US Economic Development Administration (EDA), an offshoot of the US Commerce Department, received a report that claimed there was an infection in its network. Instead of following standard best practices for identifying and cleaning up malware, they decided to "go nuclear." The agency spent millions and trashed a ridiculous amount of computer equipment to get rid of an infection that did not exist -- all of this because of bad communication and poor IT security skills.

How not to react to a malware infection

This event started in December of 2011 when the Commerce Department's IT team sent a memo to the 170-person crew that heads the EDA, telling them two of their computers were infected by malware. The first memo was vague as to how widespread the malware attack was, and the clarification that came later may not have made it all the way to the people it was meant to reach. Nevertheless, the EDA IT officer decided to go all-out in order to get rid of the infection. They employed the services of four agencies and an outside contractor, and even when they were told the malware was not widespread, they acted anyway.

According to a report from the Inspector General [PDF], the EDA's chief IT officer decided that the only way to be completely sure that all malware was gone was to burn everything down, literally. The team set out to destroy computers, keyboards, mice, TVs, cameras -- about $3 million worth of equipment. They eventually ran out of money, which is when the IT officer went so far as to request another $26 million for further recovery efforts -- denied by the Commerce Department.

The office of the Inspector General said that the EDA's persistent mistaken beliefs had "cost the government an unnecessary expenditure." Meanwhile, the EDA says that it learned its lesson, and that they had acted in an "abundance of caution."

The proper way to do things

To many of us, this may seem so ridiculous as to be laughable. But the fact remains that this is something which happened in the US government last year, not an ancient event from a time when malware removal was new, or in some small company without any proper IT crew. The fact that this could happen at all shows not only a lack of training in the actual officer who took the decisions, but also the lack of education that was provided to him along the way. An individual with a laptop who thinks the only way to get rid of a virus is to throw the machine away will likely get educated at a local computer store. But the head of IT at a government agency with millions in technical equipment may be more likely to hide his ignorance.

The proper steps to get rid of malware on an enterprise network don't require a lot of money or IT crew. When malware is detected, there are many solutions that can be used, most of them the same as what you would use for a home system. If you do have an up-to-date security software then it can go a long way to detect and eradicate the malware. To be sure your network is safe, you can run scanners such as MalwareBytes on the infected systems, and then make sure the troubled computers are clean before allowing them back on the network. If all else fails, reformatting a hard drive and reinstalling the OS from scratch almost always does the trick. If you have a good backup strategy in place, then this type of event should not disrupt your business too much.

This event shows a fundamental flaw in the system – how does someone like that manage to get into such a position? It's unclear whether the EDA's IT officer is still employed, but the report refers to him as the current CIO, so it's a good bet nobody got fired over this, at least no one at the top. Security mailing lists, logs, and messages generated by other security-related controls may be hard to comprehend for those without some basic IT training, and perhaps a manager will think he/she could get by leaving the "technical" stuff to the IT staff. But unfortunately, that clueless manager gets to decide how the money is spent – or in this case, wasted. There was evidence that the CIO ignored the contractor's advice and went on with the disposal of keyboards and mice anyway.

This whole event is both funny and sad, because it shows how ridiculous some government bureaucracies can be, but at the end of the day, those millions are real tax dollars being wasted. It doesn't take a whole lot of changes to make sure something like that doesn't happen again. Any IT group needs strict procedures on what to do when something like that happen, and the right balance must be maintained between reacting quickly, and not doing things that are either useless or worsens the problem. Identifying and removing malware is not a guessing game; there are well known ways to deal with these problems. The fact that this government officer was allowed to make up this ridiculous process shows that the procedures were either not followed, or non-existent. Hopefully, this is also something that's being corrected.

Have you experienced or heard about a worse example of IT cluelessness? Share it in the discussion!



About

Patrick Lambert has been working in the tech industry for over 15 years, both as an online freelancer and in companies around Montreal, Canada. A fan of Star Wars, gaming, technology, and art, he writes for several sites including the art news commun...

57 comments
tom_briscoe
tom_briscoe

80% of small businesses fail in the first two years. I wonder if that means the other 20% must be doing business with the government. Seems to be a surefire way to make money.

hpant
hpant

Dumping gadgets for a virus.. I wonder had this "CIO" been in medical field , he would have believed in killing off the patients who catch even common cold.


On a serious note who and how did he manage to reach to such a level is a learning for all.

RThr33
RThr33

Haha, All I can picture is a man in a suit saying "Sorry Sir, Your keyboard has a virus. We think he got it from the mouse. It's all got to go I'm afraid."

cquirke
cquirke

Potentially the worst impact of this is the loss of forensic opportunities - you've learned nothing about your attacker, and guess what happens when you re-create the original attackable state?

dantin
dantin

I can't let this government bashing go unanswered.  Who among you cannot say there have been countless incidents of cluelessness, misguided response and waste in the private sector as well.

PhilippeV
PhilippeV

Honestly, if the only response to a security event is to replace something your don't know by something else you also don't know, this is very bad.

A minimum would have been to perform some investigations to help locate possible issues (and may be find something else at the same time, that could have been more critical and that would have alerted some other services).

So when my neighbour will claim that my dog is ill, will I start erasing all dogs in my neighborhood, without even looking for council at a local veterinary ?

Ok these systems could have been isolated/shut down termporarily but why wasting them, wasting so much work time, and wasting much money to get new hardware, install it, and securing it again ? (the solution here seems worse in term of security than even doing nothing, it opens many new trap doors for infections caused by the new unsecured hardware that will be deployed).

A good security level is something that cannot be reached magically because someone decides it. It is a long term plan with many small progressie steps where many simple but efficient measures and tuning are performed, and which gains from the experience of users and administrators. This constant effort is just thrown away here. Everything is to be restarted from zero. Stupid reaction. And it will be long before the new replacement systems will be trustable again.


pgm554
pgm554

Oh ,I got lot's o wasteful gov spending stories.

A friend of mine worked for a contractor that was automating a WORM archive system for the IRS  in Fresno.

By the time the project was complete and signed off,it was obsolete as other technologies had replaced it

Couple of million from what I recall.

pipervt89
pipervt89

I would like to point out that using Malwarebytes in a commercial capacity would not be free if the government was following licensing laws. Great product, much more cost effective than all new equipment etc. Just a quick shout-out to the BSA

matt.durcan
matt.durcan

Oh bless... Rats spread the Black Death infection in the Middle Ages, so what could be more natural than to trash all the mice?? 

Or perhaps a cunning attempt to make the Taliban die laughing? 

What confuses me is how they wanted $26M MORE to replace $3M of kit.. However this story isn't all bad: WELL DONE to the business who said NO. Perhaps the IT executive could be persuaded to defect to Russia, assuming that wouldn't be construed as an act of war. 

lesobyrne
lesobyrne

What about the Irish government who spent 53 million euro on e-voting machines for local and European elections in 2004 only to find out before they were even used that they were not secure enough and that they needed to spend another 28 million to improve the security which they refused to do. So they put them in storage at a cost of 3 million a year until in 2010 they sold them to a recycling company for 70,000 euro

einaschern
einaschern

"Fail" is a verb, not a noun. Please fix the awful title.

Adam_12345
Adam_12345

Dear Department of Commerce IT staff, please, you don't have to trash your new PC or laptops. Just send me one :] and I'll cover shipping expenses.

SchusterRW
SchusterRW

Then there is the monkey business with the Transport Workers Identification Card (TWIC). This is to allow transportation workers into ports, fuel loading facilities and so on. Millions spent and no card scanners at ports yet because they can't agree on what information they need yet.

I recently had to renew my HazMat Endorsement Background Check, just so II can haul hazmat for the freight company I work for.

Decided to get a TWIC because I am considering job change. $86.5 for the background check plus fingerprints, they can't use the prints on file as all truck drivers undergo plastic surgery on their fingertips every five years.:) Then pay $129.50 so someone else can fingerprint me again on different equipment. Both sets get sent to the FBI for checking. Have to fill out the application on site because they can't access the online application and haven't been able to for some time.

When filling out the online application for the HazMat Endorsement check I get to the screen to make an appointment at my local (42 miles away) center. Shows no appointment slots available in the next six months. It is a weekend so I figure something is wrong with website. Still a no go on Monday so I call. I am told that the part to make appointments has been no function for some time. Guess they are not smart enough to put a notice on the web page to call.

But it keeps the employment rate up.

It is worth mention that the contractor for this was most helpful and pleasant to deal with.

StealthRider
StealthRider

Sounds like someone is getting kickbacks from the supplier of equipment.

No one can be that stupid. As you have so eloquently stated it isn't like this is a new issue. To assume someone in charge of IT is less educated than a high school student is an insult to our intelligence. An investigation should be launched and find out just who is profiting from this fraudulent behavior.

dave the IT guy
dave the IT guy

Not remotely anywhere close to as bad as this, but an office I worked in many years ago was double cabled. Each desk had a cat 3 phone jack and 2 Cat5 wired data jacks. When I asked about it someone said that a few years earlier someone in management was told that to move from the old dumb terminal system to desktop PC's they needed new cabling in the building. What they had was "twisted pair" and what they needed was "Ethernet" so they needed to recable the whole building.. So there were new identical patch bays installed in the main closet, labeled with the new name of Ethernet right below the old patch bays being labeled twisted pair and duplicate cat 5 drops were made to each desk...

ademenev
ademenev

Are all people commenting here really think the government is dumb? Are you sure that millions were spent on new computers?

ThaWiz
ThaWiz

Too often this happens at every level - from personal computers up through every level of business and government. How do you deal with it? Call someone who sells computers? That's why retailers have a "geek squad". Sales people posing as technicians.

david.wandelt
david.wandelt

The *true* fundamental flaw in the system is that the government has completely lost sight of it's stewardship responsibility. They think they have unlimited funds, and act like money has no value--which if they keep printing it like they have been, will be self-fulfilling!

BobaFettismyuncle
BobaFettismyuncle

Maybe that IT officer is an idiot.  Or maybe he just wanted to upgrade the entire department's equipment and used this as an excuse to do so.  Either one is unacceptable.  If a private business did that--heads would roll like bowling balls.  

We should be able to impeach government employees...

Rudkin John
Rudkin John

I'm not sure it is quite as bad, but Conficker caused a great deal of trouble about 4 years ago in the UK. I know many Councils were affected (and my own), as were Manchester Police and others.  The sad thing was the inevitability of the infection.  I cannot understand how pouring millions into fixing such problems can just be put aside as a "necessity" when due diligence would help prevent such intrusions and security violations in the first instance.  In the case of the Council in which I worked at the time there were significant outages.  I must admit to being a little smug, being the resident Mac user at the time.  We were unaffected of course.   Blackpool Council ended up with about 155 PC and laptops infected, along with 22 servers - and all because the relevant version of the security software was not installed.

  • Corporate firewall service underwent a number of restarts, causing brief interruption to some ICT services.

  • ICT Customer Support service was unable to provide normal level of service in dealing with day- to-day calls.

  • Network services suffered outages and reduced performance. This includes access to shared drives.

  • Telephony services suffered reduced performance and intermittent issues.

  • Printing suffered reduced performance and intermittent issues.

  • Council web sites suffered some interruptions to services.

  • Email services suffered reduced performance and some problems.

  • Access to the Internet suffered reduced performance and some access problems.

4. Number of Business Systems affected

Below are known examples, which show the types of impact across the Council.

  • In terms of access devices, most service users were able to use other PC’s in their offices and switch off infected PC’s, until they had been cleaned.

  • IPT phones at school sites, for a number of days, being affected by network traffic causing breakup on lines.

  • Pupil Referral Units – all sites temporarily taken off the network.

  • Significant impacts for Adult Social Services after go live with new systems on 25/11/09.

o BPM server disconnected itself from the network.
o 1/12/09 ASCIT running very slow.
o 2/12/09 Lagan could not log onto the ASCIT serversy

needed).
o 3/12/09 An F-Secure scan was run on the ASCIT database server during the day with no forewarning and the system was brought down completely. 

o All users were affected by poor performance on all applications and persistent messages from F-Secure. 

  • eGGP system down over a number of days on the intranet and Council website. eGGP also suffered from poor performance, resulting in a number of helpdesk calls.

  • The business impact at Customer First: It isn't clear in some cases whether the problems were directly caused by the virus.

o Call centre - 2 outages losing all calls.
o Call centre wallboard outages.
o Citrix outages - especially for Fylde and homeworkers.
o CPS print server infected.
o CRM eforms failed.
o CRM telephone failures - slow to start then no telephony buttons on some PCs. o Loss of voicemail / answerphone facilities.
o Normal ICT work delayed.
o Paris Payments - IVR, online and .net outages.
o People not able to use certain PCs - some are still down.
o Pericles outages, especially Fylde Pericles.
o Qmatic outages and Qmatic plasmas and LCD screens down
o Very slow log on and applications.

Virus impact on Libraries and Leisure
o Central Library on 01/12/09 - no internet access for customers or staff

o Local History Centre - some or all of the public PC’s froze

o Boundary Library - all PC’s were down all morning on the day the virus broke out. 

o Layton Library and Revoe - down time on the public access points on an odd couple of days

o The Business Administration Manager for Leisure Services says Leisure was not affected by the virus. 

This was all disruptive of services. It was highlighted that PCs and servers with out-of-date and unsupported operating systems must be replaced or upgraded. The Council still had around 400 – 500 Windows 2000 machines. Current Microsoft support dates are given below:-

o Windows 2000 for PCs and Servers – extended support period ends July 13th 2010 o Windows XP – support ends April 8th 2014
o XP Service Pack 2 – support ends July 13 2010

Is it enough to suggest that the learning is all about "we sucked it and saw" ie: when a major incident occurs and staff resources are diverted, the impact on the timescales for live projects needs to be reviewed and an assessment made of the longer term impact.

The result of the outbreak is that people became better informed about virus threats and some improvements to procedures implemented. If the lessons learned above are incorporated into the work, the risk to the Council of disruption to essential services will be greatly reduced. 

The plain fact is - this was allowed to happen in the first place, which was pretty poor really.  Remedies - well no one lost their job.  With consultancy from Synetix and overtime, ICT staff were well supported - but it will have cost the Council a small fortune in resolution, as well as in costs to remedy. overtime and external costs.

See: http://youtu.be/G03YU2Vq73c

Reality Bites
Reality Bites

saying government workers and contractors are stupid is rude to the stupid.


Government workers are by their very nature the dumbest, lowest IQ least employable idiots available out of the job market.  Who but a cretin would ever work for the lowest losers in the land?

sissy sue
sissy sue

This is another argument for small government.  If you give people access to an unlimited source of money (i.e., the taxpayer who has no power over the purse) and 0 accountability for their own stupidity, this is the waste you can expect.

bmeyer66
bmeyer66

This is not that uncommon unfortunately. I do believe that many times the people at the top in many places not just the government have not yet gotten the training, to overcome their ignorance of the subjects that they may be managing. This is not limited to IT department's only.

Zzznorch
Zzznorch

I read about stupid things like this yet I have to ask would we be better off paying one group of people to dig holes and another group to fill them in?  I look at all the waste in government and have to realize that if you slashed the Defense and Homeland Security Department budgets to the bone along with departments like Commerce mentioned above, you would have a lot of unemployed people who would then go on unemployment and welfare.  Basically paying them to do nothing and probably with no prospects for private sector employment.  If you can accept the fact that waste is a fact of government life (which it is along with backroom deals) then you need to figure out what is the best way to waste money yet still get some return.

dmm99
dmm99

The head of IT didn't get fired, and everyone got brand new computer equipment and software.  What do you mean, "How did this happen?"  There is no accountability.  The head of IT should have gone to prison for destroying valuable government property.  If he was unqualified, he should not have accepted the position.

bobp
bobp

@dantin But instances like this in private industry don't cost the tapayers money! 

In addition, anybody in the private sector who did this would be fired and never a similar job. A business run this way would quickly run out of money and go out of business. The person who ordered these ridiculous actions should not only  fired, but be sued for all the money it wasted.

bobmattfran
bobmattfran

@dantin I agree Considering that the private sector in the UK provide the system, software and consultants, the Private sector has a lot to answer for. Most of the IT  consultancy companies in the UK are gloriified accountancy companies, and have been responsible for some of the most expensive cockups. The NHS and the Air Traffic Control are just 2 of a long list of multimillion not fit for purpose systems. However, I love these so called IT companies as I make a very good living repairing the damage they cause. 

enderby!
enderby!

@einaschern Poor writing with pop culture language is very widespread. The title is fine only as a text between twelve year olds.    Maybe it is meant to be a language virus.


techracer7
techracer7

@dave the IT guy The Dumb terminals would have likely been cat-3 also and carrying serial data with potentially not all the pairs connected at the closet (if you only use two pairs instead of all four). If during the cutover to the new system people had both the dumb terminal and a PC on their desks they would need both cables as well. What looks like duplication now may have had genuine justification at the time.

stano360
stano360

@david.wandelt The problem is that you can starve the beast, but the first thing the beast does is cut off the most important parts (not the waste) because that's easier to get replaced!

dave the IT guy
dave the IT guy

@BobaFettismyuncle I think the latter of your two proposals is likely the case. Someone wanted new equipment and used this as excuse. Why else would they destroy keyboards and mice and cameras?

bobmattfran
bobmattfran

@Reality Bites Only an ignorant stupid redneck would make such a comment about people and backgrounds that you know absolutely nothing about. I could say without any foundation or evidence that private sector workers are overpaid. Same stupid comment dirfferent subject. What I can say about the private Sector consultancy from a position of both 30+ years investigating them and preparing IT evidenceof their wrong doing for prosecution, is that they are inept, greedy, corrupt and attempt bribery  in some cases in a pathetic attempt to avoid prosecution. The majority of government workers do a decent job but you are such a sad pillock you wouldn't understand as you haven't yet learned to walk upright!

pgm554
pgm554

@Reality Bites  

I did government contracting for a couple of years and I was basically doing what the IT staff was supposed to be doing ,but did not have the skill set to do.

The tragic part is ,since it's a union,your pay scale was based on years of service and these folks were making upwards of $60K back in the early 2000's.

No college degrees either.

bobp
bobp

@Zzznorch The Civil Service laws are a big problem. Before they existed all of the crooks were thrown out every 4 or 8 years. No matter how corrupt or incompetant they were, the government didn't get very big or expensive.

stano360
stano360

@Zzznorch The problem is they are not idle! They are like locusts chewing up the economy.

Reality Bites
Reality Bites

@Zzznorch ... anyone in any of the 3 letter agencies should be deported or executed for treason, that will cut your numbers down.

Reality Bites
Reality Bites

@dmm99 .... please name a single person in the USA government that is competent...... doesn't exist.

mudpuppy1
mudpuppy1

@stano360 @david.wandelt And easier to sell to a "low-information" public. Just look at the crap they pulled with the sequester. Many of them were caught saying "make it hurt so the public will agree to give us more money" or words to that effect.

bobp
bobp

@bobmattfran @Reality Bites Incompetance and/or corruption in the private sector doesn't cost taxpayers money. You can choose to not do business with a company. You can't choose to not pay taxes.

Reality Bites
Reality Bites

@bobmattfran @Reality Bites ..... Since I work with the losers everyday, sorry you are so out of line and wrong that it is comical.

How many decades did you work with IT, when a loser calls up and can't format a new drive..... he shouldn't be in charge of the IT dept.   Not one example but THOUSANDS OF EXAMPLES.

So try pulling your head in and not being a little drone for the government hacks.

bobmattfran
bobmattfran

@pgm554 A so called "college" degree is no guarantee whatsoever of common sense or application. I interview  about 10 new graduates a month. Half of them suffer from attitude problems, dressproblems and think that the world owes them a living. They find it difficult to grasp that a degree is just  one step up a very steep staircase.

Reality Bites
Reality Bites

@boomchuck1 @Reality Bites .... Never been unemployed a day in my life so suck it!!!!

When you have losers calling every day with the title of Administrator and they can't format a disk, you lose respect after a couple of thousand experiences with idiots.

bobmattfran
bobmattfran

@Reality Bites @Zzznorch You two redneck beauties would have done well in Nazi Germany, I don't suppose you wander around at night wearing pointy hats, white sheets and setting fire to large wooden crosses? I really feel that society has a major problem with ignorant, arrogant scum like you two.

NickNielsen
NickNielsen moderator

People who work with government employees are government employees...no matter who signs their paycheck.

Just sayin'...

Reality Bites
Reality Bites

@bobmattfran @Reality Bites @Zzznorch ..... so you support the TSA droolers,  I'll bet you love the DEA traitors murdering citizens, and I'll bet you cc the NSA on all your emails to save them time reading your private email too.

How much Kooky-Aid do you consume a day?

Nazi's??? you are the drone supporting the thugs..... wow you are dim.!!