Using multiple service providers to improve redundancy works only if the providers are not sharing fiber optic conduit space. Researchers suggest caution, as infrastructure sharing is common.
Every time I visit a digital service (telco and/or internet) provider, I hunt down the map showing the company's backbone infrastructure. When I find it, I ask for a copy. Like clockwork, I'm told that's not possible, and pictures are not allowed.
So when I came across the University of Wisconsin paper InterTubes: A Study of the US Long-haul Fiber-optic Infrastructure (PDF), and it had the map shown above, I was impressed. I assumed the paper's authors -- Ramakrishnan Durairajan and Paul Barford from the University of Wisconsin-Madison, Joel Sommers from Colgate University, and Walter Willinger from NIKSUN, Inc. -- were able to find the right government agency with the information. That was a bad assumption.
In the paper, the authors state the map is the first of its kind in existence. The researchers put four years of hard work into the project.
Prominent features of the map
Many of the maps and ensuing information did not include details the researchers needed (e.g., precise geographic locations of all the long-haul routes deployed or used by the corresponding networks). To work around that, the authors write in the paper, "We made extensive use of previously neglected or under-utilized data sources in the form of public records from federal, state, or municipal agencies or documentation generated by commercial entities (e.g., commercial fiber map providers, utility rights-of-way information, environmental impact statements, fiber sharing arrangements by the different states' transportation departments)."
The final map contains: 273 nodes/cities, 2,411 links, and 542 conduits (with multiple tenants).
Prominent features of the map include:
- Dense deployments (e.g., the northeast and coastal areas)
- Long-haul hubs (e.g., Denver and Salt Lake City)
- Pronounced absence of infrastructure (e.g., the upper plains and four corners regions)
- Parallel deployments (e.g., Kansas City to Denver)
- Spurs (e.g., along northern routes)
Why bother creating the map?
The researchers are concerned that the ever-increasing volume of information traveling across the US fiber infrastructure might overwhelm it. "It is either taken for granted or implicitly assumed that the physical infrastructure of tomorrow's internet will have the capacity, performance, and resilience required to develop and support ever more bandwidth-hungry, delay-intolerant, or QoS-sensitive services and applications," explains the authors in the research paper.
This may have been the researchers' original concerns, but after connecting all the dots, the team found something else.
The amount of infrastructure sharing surprised the research team. They write, "A striking characteristic of the constructed U.S. long-haul fiber-optic network is the significant amount of observed infrastructure sharing."
Infrastructure sharing occurs when multiple service providers jointly-use previously installed conduit runs rather than laying their own individual conduit in order to save money. Sharing is a common practice and understood, but until the researchers compiled data from all the disparate networks the extent of sharing was not known.
The fact that several providers are using the same conduit run has definite implications. The researchers advise, "At a high level, we consider conduits that are shared by many service providers as an inherently risky situation since damage to that conduit will affect several providers."
If a conduit with fiber optic cables from several providers is severed, clients may or may not go down. Traffic should be rerouted in near real-time. However, Paul Barford, one of the authors and professor in the department of computer sciences at the University of Wisconsin, notes in an email that there have been instances where the backup routes follow a path that includes the outage, then clients will lose connectivity.
"Our map doesn't account for how different providers route their traffic (either primary or backup paths) over the infrastructure (although this is something we are working on)," explains Barford. "The physical bottlenecks that we identified in the map can enable service providers to make better decisions about how to provisions backup paths."
Something else to consider: Schematics and logic diagrams do not take into consideration that different networks may use the same conduit. So what looks good on paper for backup, load balancing, and even security may be far from good in reality.
Things to check
If mission-critical operations are part of a company's portfolio, it is imperative for responsible parties in that company to determine whether the multiple service providers used to create network redundancy are sharing fiber optic conduit on any of the routes. If that is the case, the authors suggest the following:
- utilize existing conduits that are not currently part of that ISP's physical footprint; or
- choose ISPs to peer with such that the addition of the peer adds diversity in terms of physical conduits utilized.
The map and related data can be found at the Department of Homeland Security's PREDICT (Protected Repository for the Defense of Infrastructure Against Cyber Threats) website (account required).
- 3D printing is helping UK researchers create complex fiber optics
- Researchers smash the fiber optic long-distance record
- Researchers double throughput of long-distance fiber optics
- Google's Project Fi wants to fix your phone bill (CNET)
Note: TechRepublic and CNET are CBS Interactive properties.