Software

When your network admin hijacks your system

The situation has finally happened. A network admin has brought a city to its knees by changing the master system password. What IT leadership needs to learn from this situation. And what they need to do to cope.

Hollywood, the nice people who taught us that if you blindly pound the keys on your keyboard actual commands will magically appear on the screen, never saw this coming. Or maybe they did and didn't want to scare us with actual fact. Now it has happened.

Terry Childs, a 43 year old network administrator for San Francisco's Department of Telecommunications and Information Services is currently in jail and being held on $5 Million bail. He is accused of altering the city's FiberWAN network system to deny service to authorized users and setting up devices that would allow unauthorized service to the system.

I hope that our Security blogger, Chad Perrin will speak to the security issues in this case. My focus is on the leadership issues that arise from this situation.

What we currently know is that Childs has been employed by the city for five years. When he was hired he disclosed that he is a convicted felon- aggravated robbery and aggravated burglary in 1982. He was on probation or parole until 1987. According to a city official, Childs had recently been disciplined on the job for poor performance and was potentially to be fired.

It is believed that Childs began tampering with the system around June 20. He set a master password that cannot be overridden and does not allow for sufficient access to upgrade or maintain the system. Further, it appears that he may have enabled a third party to access data from the system that houses the 311 system, the city e-mail system, and the city servers, including confidential information. It also appears that he was reading his bosses e-mail.

I am so glad to not be this guy's manager today. Even happier that I am not the manager's manager or the person who hired this guy.

The city has brought in Cisco Systems to help them to get back in control of their brand new $3M system. They think that the bill to hack back the system may run into the millions. And Childs is being held on four felony charges. But how did it get this far?

Apparently the city hired a new head of security some months ago. She began auditing who had password access to the system. Childs seemed to not handle this well and began photographing her. His behavior became increasingly possessive of the system but he continued to have access to it. In addition, he had access to his bosses emails regarding his conduct. At least until he was taken into custody on July 13. But why did it take so long?

It is difficult to see the whole picture as events are unfolding. In hindsight, I am sure that we all agree that this guy should have been assigned to work with another system or simply put on administrative leave until the mess was sorted out. That never happened.

As the leader in this IT group, how do you go about insuring that the problem cannot happen again? How do you keep it from happening in the first place?

Childs' behavior had been increasingly odd since June 20. To the city's credit, an investigation of Childs had been undertaken but it appears that no one thought to limit his access to the system. I would have thought that would be the first step.

In the same situation, I would have taken the time to pull the employee off the system and at the least, had a chat about what was bothering him to the point that he felt it necessary to photograph his seniors and monitor their email. But there must have been earlier warning signs that were overlooked.

Regardless of what was missed and what should have been done differently, leadership has a difficult task in front of it. Beyond all the additional workload that is being shunted to Cisco, leadership needs to consider the other admins who are doing the same or similar job and consider how they can avoid a repeat of the situation. That will mean some discussion with the employees but in a manner that is not threatening to them. Certainly it will mean new oversight and new processes.

Normally, I would work with the Administrator team to define what the new processes should look like and incorporate the ideas into a new plan that we can all feel comfortable with. But given the situation, I should reconsider how much input I should take from the team. After all, a member of the team caused this problem. On the other hand, I can't blame the whole team for the actions of a rogue employee.

What would you do as the manager of this department, given this situation? Is there any "right" answer? Or is this situation truly breaking new ground?

More information:

SF Officials Locked out of computer network (San Francisco Chronicle)

Computer Engineer keeping quiet on lockout (San Francisco Chronicle)

SF city worker charged with computer tampering (CBS5)

182 comments
sarthurk
sarthurk

One fact is that most experienced Network Admins aim at becoming indespensable on the job. They do all they can to keep others at a distance, refusing to train their subordinates, even restricting their co-admins. others forget about the fact that they are employees who have been charged with responsibilities. So NT Admins, never forget that you will be held accountable someday, if not today.

meollex
meollex

Hit him hard! prosecute him to the MAX! let him be an example not only to his collegues but to all the Network Admins out there. Anyway, there are ways to reset Cisco router passwords..

mikifinaz1
mikifinaz1

Most IT people don't consider a user setting up a ghost Linux node. I usually set one up in the background (and gather all the info I need to get any where I need to go) so if some bonehead tries to steal the system I can step back in behind them. I did this once. It pissed off the IT guy but he had to grin and bear it because I pulled his chestnuts out of the fire when one of his people tried a stunt like this.

mikifinaz1
mikifinaz1

All I can say comes from a old statesman: Even the best government is a bad so the less the better and an unfortunately a necessary evil. I worked for government and if most people knew what went on in most government offices there would be officals of all stripes hanging from telephone poles from LA to NY. This is a fairly benign example of the idiots we let run our government.

waltrutka
waltrutka

i thaught you meant wonderednow!

Zpunky
Zpunky

Living in San Francisco for over 16 years I am not at all surprised. Politically, this city redefines nepotism and passive aggressive behavior. For all it's self-aggrandisement of progressiveness, it's own internal workings verge on feudalism. I would love to see Terry Childs hiring manager's qualifications, the real history and not the 'targeted' CV that was presented to HR. That this happened simply proves she was unqualified for her job. Period. I have absolutely no expectation that anyone involved in this will lose their job, except Terry Childs. That's just how things work here. All carrots, no sticks. This should be fair warning to all city administrators that independent, third part vetting is necessary for all job hires of a certain capacity or security level.

doug
doug

The real problem here is that the basic security model under both windows and unix just really, really sucks. I wonder if the mayor of SF really understands that even tho he doesn't have full access to his city's network, convicted felons hired to do routine computer chores do, and can read his e-mail and even top-secret documents from Homeland security? The whole root/administrative access model pretty much turns the entire corporate structure on it's head. Everything can be hidden from the CEO, as he can't access what other users are doing, yet the college student hired to do tape backups can spend his downtime reading teh CEO's love letters. It completely amazes me that this situation has been allowed to continue so long.

michael
michael

I would suggest this article may now be very wide of the mark. Recent reports on The Register including comments from this Sys Admin's colleagues reveal an altogether different story. They say that although he was very possessive of his system which he himself designed and installed, and he can be difficult to work with, his motive was that he wanted to protect his system from unauthorised alterations. It is reported that the system IS STILL UP AND RUNNING WITHOUT TROUBLE, and an Expert has indicated that all the talk of $5million to put everything right again is absolute rubbish. Any cost will be relatively small. It would be advisable to recheck your facts before being too dogmatic about what has happened and why

NickNielsen
NickNielsen

In this discussion (http://tinyurl.com/5l8npw), CG IT posted a link to a PCWorld article that provides some inside information. The article speculates that Childs locked people out not to cause problems, but to prevent prevent problems by not letting unqualified admins mess with [u]his[/u] network.

cactii1
cactii1

What are the chances that he could have a deal with the terrorists and has given them a key to the city? I think that if the terrorists had a key to the city's network they could disable so many things and wreak havoc throughout the whole city! This would create panic and terror throughout the whole city by the terrorists!

seanferd
seanferd

>> The Chron reported today that the public defender withdrew based on conflict of interest and Childs now has an appointed attorney. Erin Crane, was quick to call the case a "big misunderstanding"? that has been "blown out of proportion" in the media. Crane said the city's network is in no danger. "He is a very well respected computer engineer,"? Crane said of her client. She added that Childs developed the city's network and even copyrighted his work. "He worked out the bugs,"? she said. "He's not the bad actor." Rewind - he copyrighted his network admin work??

john.wang
john.wang

The cost of recovery reflects the risks undertaken by whomever is doing the clean up. Fundamentally, if the function of the system is known ie.: VLANs served, static routes and routing protocols for a switch, then there's no need to hack the switch, simply reset and or replace it and configure it appropriately. However for anyone undertaking the task they have two challenges, becoming familiar with the configuration and the potential service disruption if you miss something. I'd charge an arm and a leg for such work and I do. Ironically, there are many people in the industry that believe obfuscation is their main source of job security and ironically, management always brings in a consultant to try and extract information from that individual when in fact it's extracting usage information from the users that is necessary. There's been one constant at every company that I've been involved with, middle management doesn't know how to manage.

BillT174
BillT174

Who knows. All I know is for all the concern about access from the outside the inside is the biggest threat. Big city like this would have plenty of information for sale.

JohnMcGrew
JohnMcGrew

...as to how to deal with this guy other than throw more punishment at him. Don't get me wrong; I think they should lock him up and throw out the key. But the fact is that this guy has already committed career and legal suicide. I don't think he cares what happens next to himself in those regards. The thrill for the guy now is the fact that even though he's in prison, he's still literally holding the city hostage. They'll have to spend an unknown amount of money to regain control of their system, clean it up, and then contend with the collateral damage. At the moment, only he knows what that damage could be. Are there trojans or other back doors? How much personal data has been or will be compromised and what will the legal costs associated with that possibility be? How should the authorities deal with him? Should they forget about dealing with him and just resign themselves with manually cleaning out the system? Or will the authorities cut a secret deal with the guy to get the access back?

The 'G-Man.'
The 'G-Man.'

never forget that you will be held accountable someday, if not today?? That applies to any worker in any fold!

HAL 9000
HAL 9000

CISCO Routers/Hub/Switches as that is all that is involved here. Child's was responsible for the WAN a combination of Fiber and wire no boxes other than a Firewall device is involved in this setup. So how with a Nix OS running can you Ghost these devices? Col

terry.floyd
terry.floyd

Please take some time to read the more detailed reporting of TechRepublic's Security blogger Chad Perrin on this topic at http://blogs.techrepublic.com.com/security/?p=509&tag=nl.e102. Terry Childs was the ONLY CCIE they had on staff, and the only one qualified to manage the FiberWAN. He had been trying for months to get San Francisco's DTIS management to adopt a comprehensive security policy to protect the network he designed and maintained, but they refused to consider his ideas, even though he was far more qualified than any of his co-workers or managers. Most of the managment in this office are political appointees with little or no understanding of IT matters(e.g., Ron Vinson was a PR spokesperson for Mayor Gavin Newsom's office before he became DTIS's Chief Administrative Officer and Child's supervisor; he has no IT experience at all, and yet he was put in charge of the City's entire Department of IT Services!!!) Read some of the comments by Childs' co-workers at Wired Magazine's Threat Level blog at http://blog.wired.com/27bstroke6/2008/07/former-san-fran.html. He may have been a bit hard to get along with, but lots of my own co-workers could say the same thing about me. He was apparently a competent and conscientious security professional, and just the kind of person you want in such a position. The DA claims he has cost the city millions of dollars in damages, but all other reports are that the network is running smoothly with no downtime and no service interruptions. It seems as though Childs built a network strong enough to survive without him babysitting it constantly. Sure, there should have been some administrative redundancy so that others could handle tasks in Childs' absence, but he was apparently trying to get the City to hire a replacement or create a backup position to shadow him at the time the new Security Officer decided to discipline him for insubordination. The problem is there aren't too many CCIE certified SysAdmins in the world, and the ones who aren't already overworked are not going to take a job at a place with a reputation like the City of San Francisco.

Tig2
Tig2

Several days after the article is written. Had I a crystal ball, I would likely have written the same article in the same way. This guy doesn't deserve a pass because he designed the system. The required transparency was not present, hence his arrest. The fact is that SF may have to go to extraordinary lengths to take ownership of their network again and that is not acceptable. I used to work for a company where we referred to all pcs as Network Connectivity Devices. That is what they are. MY pc is the one I have a receipt for. I can do as I wish with it. A corporate asset falls under different rules. If I as a user cannot respect that, I should not have access to the asset. Same is true of a network. The devices currently cannot be accessed. Period. Terry Childs should not be allowed to prevail in this case, regardless of what a wunderkind he is. Failing to provide the passwords on demand should and DID get him jailed. The network does not and will never belong to him alone.

HAL 9000
HAL 9000

Just how exposed Child's is in this situation. There are reports that there was a possible breach to the system but there are no proven incidents shown. It's just a Wild Claim made to paint him as the bad guy who is doing as he pleases to the determent of others. The problem here is I'm not sure just what the defense team can do to protect him from the Incompetence of the System that he was working in. But it is a perfect example of Managers without a clue running something and not knowing what was being done under them. Of course they look good when things are going well but when the Brown Stuff hits the fan they have no idea and panic coming out with all sorts of Wild Claims none of which are based in anything factual just what is possible. Personally I would have walked long before things reached this stage as I don't feel comfortable in a position where I'm the only person to ask on any item. But as that is just me I can understand others who feel differently. Col

HAL 9000
HAL 9000

We are being Terrorized by Incompetent Bureaucrats who's only ability is to crucify the few Professional People that work there because they will not bow down and do as they are told because it breaks the ability to secure the system and do exactly what they where originally told to do. Terrorism by Stupidity is the new Catch Cry of the Paranoid. Wake up and smell the roses this isn't anything but a bunch of Bureaucrats trying to get their own way and someone doing their job. If that is any form of Terrorism then most of us are guilty of Terrorism and that is one Allegation that I am unable to accept. Col

seanferd
seanferd

Panic set in without any evidence of actual damage being done, aside from being locked out of system administration. Fait accompli, if this guy's goal was terror.

w2ktechman
w2ktechman

since it is copy protected, they cannot crack it with conventional means or he will sue them. So they have Cisco there, re-designing the entire network and avoiding copyright infringement That's gotta be it :D

jredmon
jredmon

Does dealing with terrorist work? Do we admins now have full rights to destroy company data knowing if after we have caused the company as much pain as we seem enough we can make our own deals to restore the data? How is this even a smart choice? One company might get off easier but hundreds or thousands of companies may end up with this fate if a clear message is not sent. Cause and effect.

Ragged_Scooper
Ragged_Scooper

Just like on Law and Order and many cases in real life, I am sure they will strike some sort of deal with him to disclose all of what he did in exchange for a lighter sentence.

Tig2
Tig2

This whole thing is a win for Childs and a loss for the state no matter what they do. They could lock him up forever. That MIGHT dissuade the next idiot who thinks to do something like this. Of course, it might not too. Or they could cut a deal with Childs. That might get them back into their system but then they would still need to spend a pot of money to figure out if any back doors were left behind. And potentially open themselves up to the same situation again because this guy walked. The whole thing should be forcing us to rethink security and access controls. It should force management to THINK about the wisdom of allowing people that they believe to have taken a dive off the deep end access to sensitive systems. It should encourage discussion about oversight and how it could have been used in this case. The fact that this guy has been able to bring a city to its knees is scary stuff. That isn't a bad thing if we learn something from the situation.

HAL 9000
HAL 9000

Not at all when was the last time you heard of any Politician being held Accountable for their Actions? :D It's just the Workers as Politicians or Senior Management get paid to leave and are only held to account when their actions have been so blatant that something has to be done. Even then it doesn't happen very often. :) Col ]:)

doug
doug

This is another thing that always amazes me. There's backdoors into everything. "This command, called a No Service Password Recovery is often used by engineers to add an extra level of security to networks, said Mike Chase, regional director of engineering with FusionStorm, an IT services provider that supports Cisco products."

jdclyde
jdclyde

you get that Col. Oh well. That whole "reading" thing is way over rated anyways.... ;\

Zpunky
Zpunky

I wasn't referring to Childs. I was referring to his superiors for exactly the reasons you pointed out, and pretty much stated that in my comment.

HAL 9000
HAL 9000

You have failed to take into consideration that this isn't a Company it's a Bureaucracy and as such is a completely different kettle of Fish. Common Sense or Financial Logic has no place in these places unlike in Business where everything is governed by the Bottom Line. After all when was the last time you heard of a Public Servant being fired for Incompetence? Col

NickNielsen
NickNielsen

how willing people are to believe the worst of others without hearing the whole story. They're still posting about terrorism and about him profiting from the notoriety up top. Shameful. :( And to put the lie to the implication in the original article, the SF network is not down and the city not dead in the digital water. You know it would be front-page news if it were. Edit: clarify

seanferd
seanferd

A copyrighted misunderstanding at that. What is it, hasn't the right admin with the correct security level asked for the password? "Oh, duh, we should have had Bob ask him, the cops aren't authorized!"

JohnMcGrew
JohnMcGrew

...and yet they'll probably try to do it. (That's why it will be a "secret" deal) And either way, the guy's career is shot and he's going to jail. This is hardly a route that any sane admin would want to go.

jacl
jacl

I think he wants a get out of jail card, no lawsuits, a couple million dollars for his retirement from IT field, seeing he's no longer employable. BTW, a Hollywood movie, and a couple book deals would be NICE. Blackmail REALLY may pay for this DUDE!

Brett.Blatchley
Brett.Blatchley

Hmmm, in some times and places, such people were simply made to "disappear...."

HAL 9000
HAL 9000

But because Bureaucracies encourage this type of practice dump all your work onto someone willing to do it and then hold 1 piece of paper for 5 years and you'll be busy. :D Col

jdclyde
jdclyde

And in some cases, ANYTHING connected to the network is a part of the network. He set up the network, and getting email, DNS and DHCP going in some cases (especially for a power hungry fool) could be seen as part of his job. Especially if he just DID the jobs and his fellows sat back and played solitaire. It wouldn't surprise me at all.

HAL 9000
HAL 9000

But I suppose it depends on how you identify a Network I suppose. :D Col

neilb
neilb

Now I have that bloody tune stuck in my head for the next six weeks! :_| Neil :D Hi, Cute... Passing by or stopping for a while?

HAL 9000
HAL 9000

Without any Proof it's not worth the paper it's printed on and is completely worthless. Yes he could have read his Boss E-Mail and he could have let others into the WAN here but as there is no proof of this occurring or any other indication that it's happened like unaccounted for funds exposed there is no proof or any reason to believe that it has happened. Similarly he could have wired in a Atom Bomb to trigger when any unauthorized traffic appeared on the WAN it's possible but very unlikely to be true isn't it? Why is it so important to be completely Paranoid about what may have happened and forget about what has happened here? The System is working correctly no one is prevented from processing data or moving it around they just can not access the controls of the WAN here where they will undoubtedly break things and then blame that break on Child's. These Idiots have painted themselves into a corner by their actions as if they accepted directions from Child's and they break it they will be hung out to dry for being foolish enough to trust what he said or the information that he gave them. And if they try to fix it by themselves without his help they most certainly will break it and be held up for ridicule which they deserve. Either way it's no longer in Child's best interests to help them as when they break the system he'll be the one getting the blame for the inability of the people who Crack it. Don't get me wrong I'm not saying that Child's is blameless in all of this I'm sure he's not but none of what is listed in the articles that you listed has any Real Proof that he's done anything wrong either. As for the new Security Person who was brought in to Secure The Systems I would want to see her Credentials and Work Experience before making a single comment there as to what may or may not have happened. It's just as possible that she burst into tears when she couldn't get her own way and hid in her office till Child's had left the scene. Of course that's not good press till the tables are turned to show just how incompetent that People responsible for running the show are reported to be and isn't going to happen now or any time soon. It's just not good for advertising $ to not follow the Flock of Sheep. I'm always very careful to not make any decisions based on incomplete information unlike some others who seem to be accepting that all that they read in the Newspapers is True & Correct with no possibility of being wrong. Let alone being deliberately misleading for some unknown reason. [b]In this situation who has the most to loose?[/b] That is the real question here and not anything else. Col

NickNielsen
NickNielsen

I either misread or misremembered what I read.

Tig2
Tig2

And I linked three different articles all saying the same thing. The speculation that he locked everyone out to "protect" his network is patent crap. And a violation of best practice. If he was that concerned, why didn't he voice his concerns rather than scare the crap out of the security chief brought in to develop a security plan? And what was he doing with his bosses email? I'm not ready to paint the boy innocent until some of those searching questions can be answered.

NickNielsen
NickNielsen

Thanks, Elf. Had to save that one. So how you been? Work treating you OK?

CuteElf
CuteElf

Every Bit is Sacred, Every Bit is Great... If a bit is Was-ted Cisco's quite Irate... Let the heathen lose theirs In the tangled 'Net God shall make them pay for Each IP that's a re-direct.

HAL 9000
HAL 9000

That he had allowed a third party to access the Data and that he may have been reading his boss E-Mail. Well the E-Mail bit is quite possible but the story about searching for a device to open the network in his car home and so on was just a bit far fetched for me to accept. If they had proof not a problem but the idea belief or whatever sounded way to much like the someone trying to justify their positing for my liking. But no matter there is a very Poor System and Mentality in place here that allowed this to happen in the first place. Only shows just how bad a place can be though. Does nothing to show Best Practices or even how a IT Department should work. Perhaps John Clease or someone similar could use this in a training film in how not to run a IT Department. :D Col

HAL 9000
HAL 9000

A bunch of Bureaucrats who have no idea force someone into a position where there is no way out and still retain the Security required to keep the system secure. Of course I would hand out the Master Control Password to anyone who asked no matter how stupid they where. Naturally it's perfectly acceptable not to give the Control to someone not directly connected with the system as who trusts a Police Officer with that amount of Control? But more to the point if he was to hand it out and they messed it up who would they blame then? To me it sounds as if he has been painted into a corner by the people who are supposed to protect him from Stupid Unsubstantiated Rumors and they are using this to justify their own inability to do the jobs that they are employed to do. Who better to blame but the one professional person who works there and knows what they are doing. Can't have that in a bureaucracy can we? After all it's much better to promote someone to that position who was running the hand out of Toilet Paper Squares and they will know exactly what is required to manage the system. :D Unfortunately I can laugh about things like this because I see it occurring way to often and crying over the wasted money and stupidity doesn't do anything to maintain my sanity. Col

JohnMcGrew
JohnMcGrew

...he's an even bigger idiot that we thought. The public is not that interested, and has too short a memory.

seanferd
seanferd

If this guy doesn't spend a lot of time incarcerated, the notoriety may just get him hired at a high salary elsewhere (you never know), even if he doesn't find any other way to profit.

Neon Samurai
Neon Samurai

One of the books in my to-read pile is a history of assasinations. (I get some odd books given too me) Since many assassinations are a plea for attention and "fame" of sorts by the killer, the author took the intentional aproach of discussing each case without focusing on or premoting the guilty person. Most books on the topic have to focus on the guilty party enough that they still find there place in history (if only in the memory of the person reading the book). This author has intentionally gone for the effect of denying the guilty party any further recognition while still discussiong the event. There are definately those motivated by notoriety. Look at the distructive crackers for whome it is not about getting in and out of a system cleanly; if you cant take it over, bomb it for maximum damages to show you where there. Every malicious category of people has it's truly crazy who are motivated by the temporary fame and attention that they may gain. Now, how to more frequently focus on the event and target without providing the attacker with the recognition they crave..

Tig2
Tig2

Unfortunately you are correct that at the end of this, the guy will have a certain "fame". And perhaps that in itself was the driver for all this. People baffle me sometimes and this case is no exception.

jacl
jacl

With so much publicity in this case, HE will be noticed. There are ways around the NO profiting in a felony crime thing. OJ Simpson profitted from it, WE ALL know it, his lawyers, accounts told him to move to Florida, use cash havens and pander fake memorabilias, interviews, etc. Use foreign banks, credit, "coupons", free services, e-cash, .... This usually is ONLY "enforced" on so called heinous crimes, rape, murder, etc.. In California alone, do you realize how many acts, laws etc are created, changed amended each year, few gets enforced or even followed by the legislatures themselves. They look good on paper and implementation is a totally different story.

Tig2
Tig2

I believe that there is a law that prohibits a convicted felon from monetary gain for the crime. So no, I don't think a book deal or movie is something that he can profit from. If I'm not mistaken, that prohibition extends to his family as well.

seanferd
seanferd

If convicted of whatever crimes he's charged with, can this guy profit from any media deals?

The Scummy One
The Scummy One

Hopefully, this kind of thing will not happen again. But in this case, I think that he should not get a 'get out of jail free' card. Furthermore, he should be banned from profiting for his actions (no book/movie deals, etc.). His employment was to handle the data and systems in a secure way. He violated his employment terms. If he is unemployable due to this violation, it is his own fault! Personally, I think that he should be maimed, maybe having everyone, who's trust he violated, stand in line to give him a couple of hits with a Bull Whip or something :D Maybe have a pro whip him and each person gets to toss a pail of salt water, alcohol, battery acid, or other liquid to make him scream a few (thousand) times :D

JohnMcGrew
JohnMcGrew

Instead, he will likely be put up for some humanitarian award; or will get off because of some imagined childhood trauma; or will argue that what he did was actually "performance art" making a statement about how IT has dehumanized the disenfranchised...

Editor's Picks