Fiber

AT&T fiber outage shows how easily attackers could wipe out our communications

The fiber-optic sabotage in the Bay Area on Thursday reveals a dirty little secret: Our infrastructure is ridiculously vulnerable and it only takes a few vandals (or terrorists) to bring communication to its knees.

This is a guest post from Larry Dignan, Editor in Chief of ZDNet, TechRepublic's sister site. You can follow Larry on his ZDNet blog Between the Lines (or subscribe to the RSS feed).

The fiber-optic outage---actually sabotage---in the Bay Area on Thursday reveals a dirty little secret: Our infrastructure is ridiculously vulnerable and it only takes a few vandals (or terrorists) to bring communication to its knees.

While it's unclear what exactly happened, AT&T is offering a $100,000 now $250,000 reward to find the vandals that cut into fiber optic wires and whacked everything from hosting centers---including a few of our own---911 calls and other communication (Techmeme). Sam said it best: No matter how advanced we get we're still hooked up to a big wire somewhere.

That's not going to change. The big question: How are we going to protect those big wires?

Barrett Lyon asks whether it's possible to destroy the network with a hacksaw. In a word: Yup. It happened yesterday. AT&T used Twitter---home of the Fail Whale---to communicate with customers. Anyone see the irony in that one?

What's truly scary is that we're not just talking about the Internet here. The electric grid is vulnerable. Our transportation grid is vulnerable. Our infrastructure in the U.S. is a big sitting duck. The grid and the Internet are top of mind today, but I'm reminded of the overall infrastructure vulnerability every trip into Penn Station. Every once in a while you'll see heavily armed police with their K-9 dogs in Penn Station's lobby. It's a common sight. However, if you really wanted to bring down the train station and subway it's nothing a stray backpack couldn't take care of.

Simply put, it's impossible to completely secure all of the infrastructure out there. And everyone knows it. In 2003, a student dissertation raised national security concerns. It's not rocket science to map infrastructure and cook up scenarios.

So what can we do?

Surely, it makes no sense to put armed guards at every manhole cover, fiber optic hub, power line and transportation grid. However, there may be a few things we can do:

  • Seal the manholes: It's relatively easy to pop one and clip a line or two.
  • Smart sensors: Is there the equivalent of a car alarm for fiber optic lines?
  • Better surveillance: Critical national infrastructure---like the places the AT&T fiber optic lines were cut---mostly resides in private hands. Is there a way to hook these areas up into a national security monitoring network?
  • Improved fail-over processes. Sure, the Internet is set up to reroute traffic, but it's not perfect. Can we build in more resiliency?
There is a big plan from the Department of Homeland Security about protecting national infrastructure. The 188 page tome can be boiled down to this graphic.

In the end, I'm not hopeful that any of these infrastructure assets can be secured because there's a real herding cats problem among all the vested parties. And that fact casts a pall over some of our grand tech visions.

31 comments
feral
feral

There is technology about that monitors this type of infrastructure and it is used on a number of piplines and Im sure could be adapted to manhole covers and cable runs. This one is an Australian version, yep we dont just throw shrimps on the bbq. http://www.fft.com.au/products_pipe.php Cheers

Andy M
Andy M

Although I agree with most of your points, this one raised a red flag to me: "Better surveillance: Critical national infrastructure?like the places the AT&T fiber optic lines were cut?mostly resides in private hands. Is there a way to hook these areas up into a national security monitoring network?" So if a line ran through your yard, or in the street right outside your house, how much surveillance and monitoring of your activities would you be willing to take? Of course there is always a trade off between security and privacy, but it needs to be carefully balanced. I for one do not want a security camera monitoring my house & yard, even if the purpose does not include anything to do with me. It's a short step from "we might have the data but we aren't going to use it for that purpose" to "we now have a desire to use it - and conveniently we already have it". I think there are better solutions than a national surveillance network.

HAL 9000
HAL 9000

OK while that was nasty the reality is that they where Amateurs not real professionals. If someone wanted to do real damage they need nothing more advanced than a Packet of Matches and transport to go to the right location owing to prevailing weather conditions to take out the Overland Transmission Power Lines for long distances. Something as simple as a Wild Fire can destroy miles of Overland transmission Lines that are not possible to replace in under months. One person with nothing more complicated than a box of matches and some form of transport could render a City Uninhabitable by just starting a couple of Fires that can destroy miles of These Power Lines and deprive the city of Power. While a drop off in communications is inconvenient it's hardly the end of things but take out a couple of the Main Power Feeds into a City and see just how long the Power Grid can stand the strain. With the inability to pump Petrol it's not even possible to evacuate the city. Just imagine how much thought they need to apply here walk into a store buy a Box of matches then listen to the Radio Weather Chanel to know where they need to go. Doesn't require any planning or thought really. It's all done for them by us. Feel Vulnerable yet? :0 Col

myersrl
myersrl

Anyone remember the recent TV broadcast (on CBS I believe) that talked about the two buildings on the west coast through which all communications from overseas flowed and joined with major hooks in US communications - where some certain government agency was monitoring that communication? A couple of Oklahoma City incidences would knock out major portions of all communication within the US and to those countries west. It is real efficient to have things co-located like that, but not when you talk about physical security. Especially when the electric grid hacking incidences by Russia and China are hitting the public news. Perhaps we should be incenting everyone to go solar and wind for electrical power as a national security initiative to remove our dependency totally from the grid.

heavener
heavener

I've been saying this since 2001. I see a simple scenario where a light plane loaded with commonly made explosives could destroy three major West coast cities and -- and -- take down the entire grid from the Mississippi west. I refuse to say where, but it's a sitting duck target. After all, if the grid's self-protecting function worked so perfectly that it darkened the entire northeastern when a substation overheated near Toronto, anything is possible.

NotSoChiGuy
NotSoChiGuy

...but a plane making it deep into the US shortly followed by a significant communications outage isn't giving me a case of the warm and fuzzies. I was once told that coincidences happen in two's, and patterns happen in three's. Here's to hoping for a quiet week!!

ngiacona
ngiacona

This Shows us what happen when all the eggs are in the same pot, old technolgy should always be on standby to be used as backup, Not tosed aside like we don't need this anymore...! Its Sham on us!

Jaqui
Jaqui

Remember the rolling blackouts just a couple of years ago that hit the eastern seaboard? remember what caused them? The Canadian power, and communication, systems are so closely entwined with the US that a Canadian power plant going down in a crash affects the US. hm, we have far less population, spread over far greater land area, and hitting here in Canada would screw the US still. any plan to protect the US from outages would have to include the Canadian infrastructure or there is a 6,000 mile back door to take the US system down. the entire US-Canada border. .. or, those few points on it where the lines cross. take one or two of those crossings down physically, both countries are screwed over.

The Scummy One
The Scummy One

Seal the Manholes -- yeah -- good option, then when they are needed, they cannot be used. Having them all tapped with a silent alarm would be much better. Smart Sensors? Car alarm equal? Come on now, if someone were to be doing damage, they wouldnt be hanging out there afterwards. What would this do?

boxfiddler
boxfiddler

That something like this hasn't happened sooner. This technology that we are highly dependent upon is vulnerable in the extreme. Lack of foresight, yet again. How can it be changed? It hinges on yet another dependency that is vulnerable - the electric grid. Which is a dependency that has the potential to wipe us out.

edshuck
edshuck

Financing redundancy is, as ever, unpopular. We see it in the lack of backup for operating software and data. But even when redundancy exists, the temptation is to cut costs and still have just one span, repeater, power supply or route for "only a short distance". Thus redundancy in name only. I personally did not suffer from the loss. But I could the next time it happens. My background is digital transmission, switching, network design and traffic engineering.

Jack-M
Jack-M

"Sam said it best: No matter how advanced we get we?re still hooked up to a big wire somewhere." The weakest link in the chain. Telco's call it a loop; as in: 'from the Central Office this wire loops out to the subscribers home and all the way back here'. This weakness is impossible to overcome. Whether searching for 1 particular cable or 1 pair of wires they have in common the inherent weakness of being accessible to man. Whether a repairman or a terrorist all either has to do is pop that manhole cover and drop a grenade or cut that skinny wire on the side of the building.

mjd420nova
mjd420nova

It is pretty obvious that this was an inside job. Whatever the reason for a person or persons to have done this damage, it could have only been done by someone who knew exactly where to go, how to gain access and what cables to cut. Whether the person was recently fired or this was a result of some ongoing negotiations with their unions going in the wrong direction, just about any industry is vulnerable to this kind of attack.

jpesadilla
jpesadilla

The Blackout that affected both Canada and U.S.A. a few years back was not caused by the Canadian Power Grid failing. The grid failed in Ohio due to lack of maintenance. Yes I agree both our countries are closely "connected" in many ways. There is no effective method to PHYSICALLY protect either country's power or Communication infrastructure.

santeewelding
santeewelding

Only when we trail a wire. There is no wire, or there ultimately had not be, allowing of insight, or "we".

dkoch
dkoch

About 30 years ago (way before the Internet) in a Poli-Sci class, a friend and I proposed a scenario where a terrorist/revolutionary/anarchist/garden-variety-whack-job could do serious harm to the national infrastructure by targeting the electrical grid. This was not too long after the "Grid" was improved and interconnected after the big outage in New York. The professor actually laughed at the idea as being unworthy of consideration and totally ineffective in nature. My point being, even then the vulnerability of the grid, any grid, was painfully obvious to anyone who was paying attention, academic arrogance notwithstanding. The train tunnel fire of 2001 at the Mount Royal station in Baltimore illustrated the vulnerability of the Internet when one of UUNET's major fiber trunks was damaged. It affected the entire Eastern seaboard. As in this most recent incident the resiliency of the system mitigated the worst of the effects. However, a small band of determined agents, hitting multiple targets simultaneously could have a devastating effect. This is an issue that should be taken seriously, especially since some of us have been thinking about it for a very long time.

Dr Dij
Dr Dij

it was someone looking to steal wire. they cut into nice juicy thick cable, found out was not copper and left There's alot more of those type of stupidos, including drug addicts and illegal aliens around than there are ticked off ex-employees who are willing to do this kind of thing.

ctaylor
ctaylor

10 years ago it was either Excel Switching or Nortel that had management systems available that would detect service interruptions along particular lines and set off alarms. This would allow for early detection of connection failures. If multiple lines dropped in succession (such as with a hack saw attack) alarms would notify local police authorities who were supposed to intercepted an attacker before a hack saw could cut through all of the cable.

Old-Fart-IV
Old-Fart-IV

Rather than spend the extra money to attempt to secure these facilities, why not require redundant, diverse-path, survivable links? All the data being carried on these glass strands has a huge impact our national interests (Defense, Commerce, Banking, and Security), and local services (911, Police, Fire, Government, Internet and Mobile/Landline Phone service). These services must be available ^Bwhen needed^B: 100% availability is required (although 99.9999% would be the most likely standard and it would allow for a few seconds of down-time per year). Here is one method to achieve this goal that is currently used by Data Centers and the Telco Carriers: Service providers are required by Service Level Agreement (SLA) to have 100% availability via divers routing capabilities in-place when the contract is signed; this include total diverse routing of the circuits (no single point of failure) with dual entries into switch/data/host facilities; and if a failure occurs that takes down both diverse paths, there could be big penalties for lost business. Back in the days when I did Fiber work for the Government, I've been to the scene of these types of outages. Usually the person running the equipment (backhoe, cement saw, trencher, etc) was unaware of what havoc they had caused, but knew something was wrong when Police and Government people arrived (we were transported to several incidents via a Helicopter). In those days we guarded the fiber links using tamper / disruption monitoring of spare "dark" fibers within the cable (with 8 to 16 fibers inside the cable, 1 or 2 were reserved for testing and monitoring). Any cable movement or pressure on the cable caused an Alarm to sound in the Ops Center and by using an Optical Time-Domain Reflectometer (OTDR) the location of the incident could be determined within a few meters (if the total cable length was less than 5 km) and within 10-20 meters on longer lengths. I also remember several massive outages affecting traffic (voice, data, web, etc) around y2k in Southern California. This outage near SF in Sept 1999 shutdown my employer's world-wide data & voice network until we could get the carriers to re-route our data via Canadian circuits. http://findarticles.com/p/articles/mi_m0CGN/is_3758/ai_55943308/ Until users demand more reliable and survivable service, the larger capacity optical cables that could be damaged by small incidents will impact more people in more ways due to the growing number of devices we use in our every day life and work (Cell Phone, Mobile Email device/Blackberry, WiFi, etc) Be on the lookout for: Fiber Seeking Backhoes (if there is optical fiber in the ground, the hoe will find it and break it) and Cement Cutting Saws (if there is optical fiber under the concrete, it'll be found and severed) Edited -- because I can :) JR

Data Nut
Data Nut

This may seem low-tech or naive, but it's the main reason I don't want VOIP. Ok, the phone lines are still wire, and mine is fiber optic now, but it's still an alternate means of communication. I still have my old CBs in the garage too. Don't know if they work anymore, but it can't hurt to have this stuff either. But it doesn't stop at grids and fiber lines, folks. Look at our financial crisis - a few really big banks doing riskier and riskier things - when one falls it takes out all the others because they're too big and too intertwined. It's a house of cards. We as a society have to stop consolidating everything. Why do we do it - to squeeze extra pennies out for the stockholders? Who's really getting rich on those pennies? Not the average joe. History has shown what monopolies can do. Having just a few really large players is just as bad, it just looks better on paper.

dcolbert
dcolbert

Really, this is clearly a response to AT&T's controversial telecommunications policies over the last few years - Data traffic rerouted to NSA computers for warrantless analysis, institution of caps on bandwidth usage, playing to the interests of the RIAA and MPAA. Anarchist techs (in San Jose? *Unthinkable*!) clearly wanted to give AT&T a little of their own back. Am I celebrating these terrorists and vandals as modern day Robin Hoods? Of COURSE not. (In some small part because I have a certain fear of having Corporate Sponsored Government Thugs wearing jumpsuits with a TLA on the back kicking in my door at 2 a.m. and putting on the rubber gloves before they "question" me.) They put lives at risk through their actions and should be held accountable. At the same time, I think an analysis of the conflict that led to this crime is important. I highly doubt that these were Middle Eastern Islamic Extremists doing a dry run on a larger attack designed to disable key U.S. infrastructurer. But the U.S. government is certainly willing to leverage such an act to increase fear among the civilian population by manipulation of the press. A civilian population and press that is oddly naive and easily duped. Wasn't the Internet born of a DARPA plan to make a network infrastructure that could survive a nuclear war? Hasn't the tech press talked for years about how this system has worked, how the network sees censorship as damage and reroutes around it? China has struggled with preventing citizens from accessing subversive sites for two decades now, and a few twenty-somethings with bolt-cutters and hack-saws are suddenly a threat to the communication grid? I'm kinda doubting they're that much of a threat to the electric grid with a *hacksaw*, by the way, although I'd like to see the Ogrish.com video of them trying... My real point is, this is a non-story. Are we vunerable? Of course. Safety is an illusion, just as is civil society. Don't believe me, just ask anyone who lived through Katrina how quickly that surface facade falls away when things get flakey. Sealing manhole covers and putting better sensors on miles and miles of infrastructure are Feel-Good measures that add a ton of expense (that gets passed on to us, the consumer), but doesn't really achieve anything. "Locks keep the honest people honest". But, *honesty* is probably what keeps most honest people honest. Locks give you a false sense of security. The honest guy isn't going to even bother trying the door, and the dishonest guy isn't going to be dissuaded by the lock. As far as terrorists, we collectively forget, 9/11 wasn't their first attempt. The problem with THAT enemy is that they're so darn persistent. "If at first you do not succeed..." Again, sealing man-holes and adding CCTV or having a guy with a dog walking the station isn't going to stop those guys for long if they get something into their mind. Finally, this affected a city, a single city, and not one of the most MAJOR of metropolitan areas near the affected city. Additionally, a city full of technical know-how, expertise, and experience. Extrapolating that to a nationwide, synchronized attack on our entire national infrastructure that could somehow maintain a shroud of secrecy from our very active network of Post-911 espionage spooks seems unlikely to me. I'm not buying this - it is media hype and sensationalism. More yellow journalism. Although maybe the author is sincere in this post. Just because he is a journalist, doesn't mean his critical thinking skills are developed enough to see through the slippery slope on which his article is built. Maybe he is buying into the hype, and being sucked into the culture of fear.

lee.wilmeth
lee.wilmeth

Ironically, we created a $50 Billion a year Homeland Security Department, and they can't protect our vital infrastructure from a simple vandal. Yet, they constantly hassle us at the airport over a 4 ounce bottle of shampoo. They need to stop playing PR and actually start using the vast amounts of money we waste at Homeland Security to actually implement some "homeland security".

wayoutinva
wayoutinva

And the blackouts only stopped rolling when it hit areas of the infrastructure that were actually kept up (i.e. maint was done them as needed).. One of the reasons I heard was (dont know how true it is) because of the higher labor costs up north, the facalities put off doing required maint. to save money...

Jack-M
Jack-M

The typical manhole is about 10' by 10'. The cable comes in from a conduit, it's placed so the necessary work can be done and out the opposite conduit it goes. The most cable you could get is, maybe, 40'. That's what a typical copper thief gets. A terrorist on the other hand, with 1 or 2 grenades can disrupt communications for 10's of thousands. Not a pleasant thought.

Datacommguy
Datacommguy

I don't think there's any question that it's pretty obvious our power and communications grids are vulnerable. Unfortunately, even back when a good part of those grids were being designed and put together during times when there were real threats - as far back as WW-II, the cold war, and even post 9/11 - it was evidently considered too expensive to "do it right the first time". And fixing it now would take more money than anyone is willing to spend. Like New Orleans which certainly knew that some day they'd have to pay the price for inadequate dikes and levees, the cost of doing what needed to be done made it all too tempting for those responsible to ignore the risk and put it off until it was somebody else's problem. And until something the size of Katrina calls attention to how fragile the communication and power grids are, we're going to remain vulnerable.

aeiyor
aeiyor

I've read through many of the replies referring to the primary article. I actually am in agreement with a lot of what you've shared and mentioned dcolvert. Though I believe it covers more ground than some of the postings refer. Also just as a side note, you particular background is interesting because I also was in MCI Communication around the same time - working on the very same things you mentioned within your profile... so I believe I know you from our past. :) Back to the topic at hand... It's pretty amazing how many of the movies and books out there cover similar issues like this and how not so far off they are... though they are based at this time on fiction... I relegate it to "at this time" because like a lot of fiction (especially science fiction) from today is factual later on. I believe the heart of the matter covers a few things.. 1. We have to get past our dependencies within our culture and society as these dependencies hinder us. By getting past the dependencies I am not meaning to do without but establish means of various support such that if something is not available - alternatives are. 2. We need to get back to interactions with people and our surrounding communities in cohesive bonds. Staying in societies where we are segregated, separated, apart and isolated means we are vulnerable. United we stand - Divided we fall. (I know that was a song from way back when...) 3. We need to develop more awareness, understanding, discipline and consideration for: our environment, technology, government, people, ecology. 4. We also need to develop the responsibility to understand the parts we play in our own and everyones lives. There was something mentioned from Jurassic Park which I feel applies to technology in general and not just genetics.. (quote from Jurassic Park)... "...I'll tell you the problem with the scientific power that you're using here: it didn't require any discipline to attain it. You read what others had done and you took the next step. You didn't earn the knowledge for yourselves, so you don't take any responsibility... for it. You stood on the shoulders of geniuses to accomplish something as fast as you could and before you even knew what you had you patented it and packaged it and slapped it on a plastic lunchbox..." I believe this applies to a lot of the technological advancements we've incurred within the past 100 years. Anyways just some added thoughts and meanderings on the potential things we face. I don't subscribe to know everything within this matter only some potential insights to what transpires and what may take place.

Support Slug
Support Slug

Well said, Mr. Colbert. The only thing we have to fear is fear itself... and apparently rubber gloved thugs. The Bush administration may not have perpetrated the attacks on 9/11, but they have certainly taken full advantage of the situation in order to have their way with the Constitution and our liberty. Our current administration seems to be continuing up that same path. When we have given up all of our freedoms for the illusion of security, the terrorists will have won.

Jack-M
Jack-M

To address your last point first. Manholes are extremely dangerous if proper procedures are not followed. There IS a way to lock the lids down. Not foolproof but enough to slow bad guys down for a while. In the case of some type of attack on our communication system the holes near a vault should be doubly protected. As far as the amount of copper or fiber in a manhole there is usually plenty if room (in most) for all the wire and cables PLUS a splicer and his helper, a place for both to sit, fresh air and going way back a solder pot and all the tools necessary. I think thieves are generally too lazy and fastidious to plunge into a muddy, silty, dark unfamiliar underground space. Unless of course they're comm. workers and know what they're doing (HA). JackM

dcolbert
dcolbert

That one person's terrorist is usually another's "revolutionary" - and where terrorists and revolutionaries start cropping up, that is a sure sign of injustice, tyranny, corruption and oppression. Which makes me worry more that this was an INTERNAL, domestic attack on America's infrastructure far more organized and politically motivated than some white-supremist, anti-federalist rednecks blowing up an FBI building with a truck full of fertilizer. It almost has a "V for Vendetta" quality to it. Now, if so, the question is, does our government perceive it as a desperate wakeup call from her citizens, or a challenge to her power that must be ruthlessly crushed? Guess where I'd put my money? The population will remain asleep, and the Government will continue to follow the same course it has for the last 8 years or more. FWIW, I think that all this talk about civil liberty, ideals of the Republic, personal freedoms and just government are too little, too late. I think the game has been lost, probably for decades.