Networking

Who is really to blame for the San Fran network lockout?

A strange sort of techno-drama is playing out in the city of San Francisco, California right now. The blame for the fiasco may not be as easily assigned as it first appeared.

Last Sunday, the 13th of July, a computer network administrator named Terry Childs was taken into custody by police and charged with four felony counts of "computer tampering." The San Francisco Department of Technology has a new FiberWAN, and Childs was intimately involved in its implementation. In fact, he is apparently the only person in the city with administrative access to most of the Cisco routers on the network -- and his unwillingness to turn over the administrative password is the reason he's now in jail, unable to come up with the $5 million he would need for bail.

Beyond that, the story gets a little muddy. The DA's office is keeping a lot of what's going on under wraps. The press doesn't have free access to Childs; his lawyer must of course be careful what he says; city officials aren't talking much; and Childs' supervisors and coworkers aren't rushing to tell their stories either. There are quite a few articles reporting parts of the story:

What really happened -- and who's really to blame?

The following tale is my view of what happened, pieced together from the above and other sources.

In the beginning . . .

Terry Childs was a smart, talented network engineer, an indispensable resource for the City of San Francisco Department of Technology. He was also not the friendliest man on the planet, with a touchy temper and perhaps a bit arrogant -- and had a criminal record, convicted 25 years ago in Kansas for aggravated robbery. He had a low opinion of some of his colleagues and his non-technical supervisors. Despite all this, his skills were respected, and some -- including Mayor Gavin Newsom -- described him using terms such as "well-liked." When the lead network architect for rolling out the new FiberWAN wanted to make sure things worked, he relied heavily on Childs' expertise, and, in fact, Childs ended up performing most of the implementation for the new network.

To ensure the continued stability of the network, he took on the task of maintaining it almost single-handedly, having a great deal of difficulty trusting any of his colleagues to do network administration tasks for the FiberWAN without screwing something up. His bosses, colleagues, and intra-organizational clients understood that someone who did not know the network sufficiently well could innocently do a lot of damage, and were mostly content to just let Childs handle it. His local authentication system with sole access to administrative functionality in his own hands had been in place for months, if not years, and in the words of an anonymous source inside SF's Department of Technology, "everyone more or less accepted it."

Childs was frustrated with his circumstances, however. Among his many problems as the lead network engineer for the entire city, he complained that his direct superior was "intrusive, incompetent, and obstructive"; the managers above his direct supervisor "had no real idea of what was going on, and were more interested in office politics than in getting anything done"; he was overworked, putting in far more than 40 hours a week (receiving comp time for overtime work that he would never have time to use); "many of his colleagues were incompetent freeloaders." Apparently, there was some truth in a lot of his complaints, and personally I'm not surprised at the thought that they might all be true.

Things come to a head

Things came to a head when a new information security manager was brought on board by the City, and Childs came up for a performance review. The security manager started prying into things, trying to get Childs to give up administrative passwords for the network, and in his review he was told he was performing poorly. It quickly began to look like Childs' professional head was going to be on the chopping block before long. Considering his circumstances, it seems reasonable he might be flabbergasted to discover that he -- the only person willing and able to do much of what he did for the city every day, the man who kept everything running smoothly, who was effectively on-call 24/7/365 and put in as much overtime as anyone for effectively no reward at all -- was on the fast-track to being fired for poor performance.

Part of his frustration revolved around the fact that, according to the anonymous source from inside the Department of Technology, Childs had told him, "I've been trying to get them to approve [a security policy] for years. I've written ones up and submitted them, but they don't want to do it, because they don't want to be held to it." Finally, the brand new security manager has put a policy in place -- a policy that is unenforceable, essentially just an unmodified template from the CCISDA that's still awaiting discussion and alteration by a committee that doesn't yet exist.

He was dismissed from work for "insubordination" on the 9th of July, but still received his regularly scheduled $127,735 a year salary paychecks. He may very well have become verbally confrontational with his superiors and the new security manager -- the latter of which became so disturbed after an encounter with him that she locked herself in her office to escape having to deal with him. He is alleged to have begun monitoring others' communications over the network with regard to his personnel evaluation case.

He refused to give up administrative passwords. He was threatened with arrest and continued to refuse to give up administrative passwords for the FiberWAN routers. Finally, he was charged with four felonies, and now sits in jail, with a bail determination five times what you'd expect many murderers to receive. As of Tuesday at the latest, according to his lawyer, he has been willing to cooperate -- but the DA's office refuses to comment on talks with Childs and his lawyer.

Officials suggest there may be reason to believe Childs gave access to some third party, outside of law enforcement, the DA's office, and the city's Department of Technology, and even said he might possess some kind of "electronic device" that might be used to gain access to the network and destroy "hundreds of thousands of sensitive documents," presumably including jail bookings and other law enforcement documents, payroll files, and e-mails. These bold assertions of the danger Childs represents were followed up by searches of his home and car for devices that may be used to compromise network security, which turned up -- nothing.

According to the mayor, "There's nothing to be alarmed about, save the inability to get into the system and tweak the system. Nothing dramatic has changed in terms of our ability to govern the city." By all accounts, the entire network continues to run smoothly, in Childs' absence -- perhaps because nobody who doesn't understand the workings of the FiberWAN configuration can change anything.

The blame

It's obvious that, officially, Terry Childs is taking all the heat for this. Based on what I've read, and the mental picture I have of the situation (as explained above), that's not really a fair assessment of the situation.

  1. Terry Childs treated the network like his own personal kingdom. That's not the best way to deal with such things, by any means. He should have documented everything, created additional administrative passwords and procedures by which others can access them if he gets hit by a bus, and otherwise done what he could to make sure that the sanctity of network performance, stability, and security didn't rest entirely on his shoulders. It's not an uncommon state of affairs in the IT world, though, and at times is even a necessary state of affairs when a network administrator has too little power and too few resources for the responsibility he must assume.
  2. His superiors gravely mishandled the situation, obviously. Even if only half the story told by the anonymous source and only half the implications of what has been offered in the official stories are true, they've done everything wrong from one end to the other. It's possible the only reason things have worked as smoothly as they have so far is Childs' skill and dedication.
  3. The prosecutors and law enforcement officers involved, judging by what I know of prosecutors and law enforcement officers (to say nothing of human nature in general), are probably more interested in convicting Childs than they are in resolving this matter. If he's really willing to cooperate, and if the whole matter was the result of a misunderstanding (as Childs' lawyer contends) and mutual mismanagement, most of the felony charges against him should be dropped and the passwords recovered. As long as Childs maintains his innocence and refuses to plead guilty to several felony offenses, though, it's likely the DA's office will do more to hinder attempts to get access to administrative passwords out of Childs than to help.

As far as I can see, there's blame all over everyone, like someone filled up a gigantic balloon with the stuff and everybody involved stabbed it with a letter opener at the same time.

The solution

It's possible I gravely misunderstand the circumstances, given the fact I surely don't know all the important details, but considering the way these things tend to play out, I find it unlikely that they diverge much from my guesses above.

If I'm close to the mark, there's a simple solution: dismiss any charges against Childs except those related to obviously illegal and unethical behavior; keep charges related to actions such as monitoring others' e-mails without authorization; and reduce the severity of the remaining charges. Then let him go with probation in exchange for completely divulging all information necessary for managing the network, including passwords, backups of router configurations, and so on.

Then, when all's said and done, let the Department of Technology suffer the consequences of firing the only man in the city who could do his job and siccing the police on him. They've certainly made an uncomfortable bed for themselves.

About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

146 comments
dcollins
dcollins

His bosses need to be sitting in jail with him. It is negligent to allow someone to be the sole gatekeeper of the IT information. They allowed him to run the network in this fashion and forced the situation to stay this way by relying solely on him. It will probably cost a ton to recover from being "locked out" of their own systems, but they deserve to pay if he is to be held in jail. Having been the system administrator of a bank, and now the manager of a large IT support firm, I can say this scenario is not uncommon.

ernestm
ernestm

So Childs is just a garden variety narcissist/while collar criminal. But the real problem is that management let anything critical be in the hands of just one person. We see this in our enterprise. Everyone wants to "run lean" (not hire people), and there's a decent percent of people you do have that need to go (and there always seems to be obstacles in the way of doing that). So you get the "one guy" who is the only one who knows how to do technical thing X. Managers should insist on cross-training and sharing - the most trivial example being passwords, but any significant technical area. Plan your servers N+1 and hire N+1. Make the risk clear to upper management that if you try to get away without putting the resource and thought into your work distribution, things like this happen.

seanferd
seanferd

SF Reveals Usernames And Password To City Network In Accidental Effort To Prove Terry Childs' Case For Him San Francisco DA discloses city's network passwords The San Francisco DA has made public 150 usernames and passwords found on rogue IT admin Terry Childs' computer The Deep End | Paul Venezia Quickie on the SF Public VPN passwords story Finally, some actual information on the Childs case The anti-admin stance

ellsanto
ellsanto

George Bush is too blame. Him and his war for oil. what a jerk.

j-mart
j-mart

After having worked in two different government departments earlier in my career, "empire building" is the normal mind set of most long term employees.Total control, never let go of all of the cards, make yourself indispensible not by how well you work but by how much important information you can horde and control, standard way of doing business. It would not suprise me if this is all due to a clash of empire's and the struggle to be in control.

mikolid
mikolid

Working with ball hogs on the team is not conducive to the overall success of the project. Investing in the success of coworkers is an important aspect to personnel success. Good leaders do not have to worry about sharing with someone who will try to back stab their success because good leaders cannot be taken down. All "Childititions" (new word) should be fired as early as possible to nip the problem in the bud.

melekali
melekali

That sounds like a expected scenario. Considering the dude kept everything running and was doing so on his own, it's no surprise no one else has authentication information for this FiberWAN. Then they put incompetents in charge and they question the expert as if they know more than him. That's just irritating. While he shouldn't have done the illegal stuff like monitoring communications, when I consider what they were doing to him (wrongly, I believe), he needed to be able to defend himself against this obvious lynch mob. He ought to be given his job back and a raise and required to document everything and all charges should be dropped.

hcombs
hcombs

I had an identical situation when I joined a small company as IT Manager. The lead Sys Admin worked from home, refused to come to the office, ever, or even meet me for lunch or after work. He had designed the network from the ground up and treated it like his private domain. He refused to give any other admin or even the CEO the passwords to the networking gear. We discovered he was working full time for a local ISP as a Sys Admin and moonlighting as free-lance contractor even though his contract with our firm required a minimum of 8 hrs per day. He saw any management requests such as a network diagram or sharing access to some devices as meddeling and intrusive. He was rude, argumentative, and refused to file any change control documents or complete trouble tickets because "documentation is for loosers". I discovered that his refusal to follow direction and frequent telephone insults had resulted in the resignation of my predecessor who could not take the frustration. I pleaded often with the CEO to let me fire this man and get professionals in to recover control of the network. The CEO refused to let him go because he was terrified that this man would destroy the network out of spite. When I did bring in Cisco to try and help us get control of the network, he refused to cooperate and complained long and loud to the CEO about my meddling with "his" network. After six months of this it came to a head and I was asked to leave because I couldn't "control" this admin. I have been there, and I have NO sympathy for the selfish, egotistical, and above all unprofessional behavior of Terry Childs.

Plant Doctor
Plant Doctor

This is going to make a wonderful Dilbert series.

navaneetham
navaneetham

I agree with Chad Perrin, for his perfect thinking and the conclusion. The whole situation in this case is a bad management, and the trouble started by the new Security Manager, handling the one man doing network management.

HAL 9000
HAL 9000

Part way locked out from was the optical Fiber backbone WAN that was designed, built and Maintained by Child's here. They where not locked out from any form of data Storage device nor where they prevented from sending that data wherever it needed to go over the WAN but they could not see what the WAN was actually doing to see if it was cloning the data and sending a copy elsewhere which apparently it wasn't. About the only people to directly suffer in this entire exercise is Child's. The Management that was supposed to make life easier for him to do his job seem to be held up as the hero's here and they are the ones directly responsible for this mess. To that end they consistently refused to allocate staff to help Child's or to set a Security Policy. As Child's was supposedly a Professional and knew what he was doing he didn't need to waste tax payers Money in having a bunch of Bureaucrats draw up a security Schedule. Personally here if Child's deserves Jail for his actions I personally think that all of his direct Supervisors need to be shot and the remainder deserve to be granted Life Imprisonment for their Involvement in this fiasco. That includes everyone involved up to the Mayor here I might add. Col

Neon Samurai
Neon Samurai

But each possitions task list tends to get in the way of training time or the required practice after training that locks new skills in place. To complicate matters, the other's can't do what I can do with my tools. One has the best chance of figuring how what crazyness I've built but equally, I can't do what they do with databases though I'd have the best chance of figuring it out. The ideal is having everyone on the team of equal technical skill level and cross trained on each other's tasks but I've never seen that outside of complete groups trained from zero to skilled together.

The Scummy One
The Scummy One

While they are claiming that Childs MAY have used it or may use it in the future to impersonate someone else, then they go and state that it is a phase 1 PW, and someone cannot gain access without a security device.. Who are they trying to fool? If it requires the part 2, and if they have found no evidence that he did otr had intentions to log in as someone else, and/or was seeking to replicate the part 2, then their statement is just cr@p, and they are again, making wild assumptions due to the general publics knowledge.

HAL 9000
HAL 9000

And about as believable. Some of the claims are just unbelievable if they are to be though of as True and Correct. The rest pertaining to the actual WAN Network are exactly what the Design Engineer should have. It's far more believable that Child's is incompetent for not having this documentation. The bit about listing User Names & Passwords beggars belief and only proves complete Incompetence on the part of the City Officials who seem Hell Bent on proving their case with Wild Impossible Allegations. When it's all said and done it looks like Child's is being accused of doing his job here. :D So if he's guilty of these Offenses all I can say is I'm glad I'm not in the US as this is Insane. Col

jdclyde
jdclyde

that you are stupid. What a Jerk.

w2ktechman
w2ktechman

how is GWB to blame for a netadmin and his managers in this case? I really am eager to know! Or are you just putting in a Bush slam cause you can?

melekali
melekali

I don't do that. I make my name by my demonstrated abilities. I fix shit and they like that.

HAL 9000
HAL 9000

The Bureaucratic Mind Set is a very interesting thing to watch isn't it? :D Better if you are no where near it when it implodes but it's still interesting to watch the complete waste that is generated by it if you can stand the stupidity that is of course. :0 Col

OnTheRopes
OnTheRopes

"But there was a snag, Ballard said - the code that Childs supplied to Newsom didn't function immediately. Newsom had to call back the attorney, who provided more information, and the system started working, officials say."" I immediately thought, "Caps Lock". :^0

w2ktechman
w2ktechman

and lack of policies, including who the password should be given to, according to the article. Childs claims that when management had the password, they wreaked havoc over the network, including introducing virus. The last paragraph speaks for itself "Referring to the felony computer-tampering counts, Crane said, "Mr. Childs intends to not only disprove those charges, but also expose the utter mismanagement, negligence and corruption at (the Technology Department) which, if left unchecked, will in fact place the city of San Francisco in danger."" Hmmm, It looks like it may be a bit more interesting coming up soon! Stay Tuned!

seanferd
seanferd

Different if there's a ball-hog in a room full of players. It doesn't sound like there were any team players there, at all.

apotheon
apotheon

Each sentence in your post could be separated from the rest of the post and printed at the bottom of one of those motivational posters.

HAL 9000
HAL 9000

[i]The whole situation in this case is a bad management, and the trouble started by the new Security Manager, handling the one man doing network management. [/i] The [b]Bad Management[/b] is certainly correct but it started long before the new Security Manager came Onboard. The person who designs the system should never be placed in a position of being required to Manage it after it's setup. That's Abysmal Management and even worse Work Practices, it should never of been allowed to happen under any circumstances and it would only happen in a Bureaucracy where you have Bureaucrats who think that they know it all and have no one controlling them to protect them from themselves. That of course covers everyone in this situation including Child's who has contributed to this mess in at least a small way. Col

w2ktechman
w2ktechman

Hmmm, just wondering how much Childs can sue the city for if he didnt really do anything wrong :0

The Scummy One
The Scummy One

I think that everyone deeply involved should be replaced immediately! There, let the dumba$$es be 'freed' from the burden of thought, and replace them :D Hmmm, who would wanna work there now though??? ?:|

eric
eric

. . . and getting a response! Quick! Somebody call a Doctor(ow)!

melekali
melekali

...let's blame George junior and impeach him for that as well...:o)

jdclyde
jdclyde

If someone were going to try to make a slam, you would think it would not make them look even stupider than the person they are pathetically attempting to slam? Clearly not in this case.

HAL 9000
HAL 9000

You got Promoted in the Civil Service? yes I know a lot of people like that in the Public Service and they are still in the same position 25 years after they joined the place. Sure they have a different Title and more money but only just above the CPI increases but they are doing exactly what they where when they started while the Incompetent ones have been promoted as high as it is Humanly possible to go and have retired on the Top Super that is available to Public Service. I've yet to see one of the workers get treated that way by the ones who are responsible for supporting them in their work though. :D Col

j-mart
j-mart

but I have a co worker who spent much more time in this enviroment, works hard and knows what he is doing but tends to hold on to his security blanket by keeping details of whatever he is working on to himself, which leaves us looking stupid when a customer wants basic information when he is not around.

w2ktechman
w2ktechman

I guessed that someone forgot to write down a Cap!

HAL 9000
HAL 9000

I just love the idea that he the Ultimate Boss was given the correct information and didn't understand the importance of the finer details though. :^0 Col

HAL 9000
HAL 9000

[i]In her motion to reduce bail, Crane said Child's had been the victim of a "bad faith" effort to force him out of his post by incompetent city officials whose meddling was jeopardizing the network Childs had built. At one point, she said, Childs discovered that the network was at risk of being infected with a computer virus introduced by a colleague. "Mr. Childs had good reason to be protective of the password," Crane said. "His co-workers and supervisors had in the past maliciously damaged the system themselves, hindered his ability to maintain it ... and shown complete indifference to maintaining it themselves.[/i] No that wouldn't be complete indifference to the smooth running of the palce or sure signs of a Us & Them Culture in that place where they are more interested in Playing Politics rather than doing what they are paid to do by the Management at the very least would it? :D I like the idea of handing over the codes to the Boss who misunderstood the importance of something so it required to be repeated so that the Team could look in to see what was happening though. Or the Fact that the Mayor is now a Witness for the Defense and ultimately the only person that Child's could in all honestly pass the Codes onto. Now if the system fails the Mayor has to wear the responsibility for the incompetence of the staff appointed to run the show. Should be an interesting Time and the Court Battle should be very enjoyable once it gets on Prime Time TV. :D Col

seanferd
seanferd

Continuing "Security Infotainment", indeed.

seanferd
seanferd

Have one. No LOLcats allowed.

mikolid
mikolid

I had to laugh when I thought of how the posters would look. Colorado rules!

seanferd
seanferd

In the event of a real emergency, you would not have seen all this clowning around. Except in case of a hurricane.

HAL 9000
HAL 9000

Spend 44 Billion + Cost Blow outs to save the 4.5 Million that was the cost of the Optical Fiber Cable that I suggested that they roll out years ago. At the time the Higher Up Decision Makers insisted that as they owned Telstra they owned the cable even though there was talk about selling off Telstra back then. My comment that Telstra would steal their Bandwidth and use it for their own needs and it wouldn't be available when required was ignored with the standard reply we own them they will do as they are told. Well they got exactly what they wanted saved a few $ and now don't have the required Bandwidth available and to make things even better have to not only bring in all the equipment to lay cable from overseas but currently have Geologists scoping a route for the new cable because can't run it beside the existing one without running the risk of digging it and breaking it. But they did save 4.5 million so they did the right thing. :^0 Col

HAL 9000
HAL 9000

But of course there is Complete Incompetents in hiring the people who didn't know or understand what was happening in this case or worse still expecting that this hooch pooch was a [b]Acceptable Standard.[/b] But of course their Budgets looked good. :D As they say pay peanuts and get monkeys but never expect the High End Pro's to understand what has happened easily. :0 Col

seanferd
seanferd

But no one asked for port numbers, now did they? Now that's incompetence! :D

HAL 9000
HAL 9000

He just read the Alpha Numeric String and didn't think about Case at all. :D Col

Neon Samurai
Neon Samurai

Damnit, the breaks are not working. There are three peddals and I keep hitting the one one in the middle.. and what's this stick with all the numbers writtein on the top if it is a funny looking number four shape? hehehe.. this could go on for days.. ok,.. minutes at least but I'm easily amused this week. ;)

jdclyde
jdclyde

but we used to have an engineer that ran over the parking block and into the building. He simply forgot to hit the brake. :0 He WAS the real life version of the absentminded professor. ;\

w2ktechman
w2ktechman

Oops, how do I use the brakes now? :0 Is that a wall in front of me??? LOL

Neon Samurai
Neon Samurai

.. and here is the key to the driver's side door. Now, for the love of baud, leave the parking break on and don't touch the shifter until you make sure you've adjusted the mirrors and know what the dohickies do. ;)

seanferd
seanferd

What they'd need to fix the system after he gave up the pwd and the other staff got into it.

Editor's Picks