Grey Lady Down
True Stories:
About a dozen years ago I worked for a major Western Retailer, located in a fairly rural location about 180 miles "inland" of Denver. To make sure their web facing eCommerce servers were "on the ring" they co-located their eCommerce servers in San Jose California.
One day our CEO walks into the eCommerce Department claiming our e-Commerce site (and millions of dollars a day in sales) was down. This just as we began receiving numerous calls of lack of Internet connectivity throughout the building.
A quick review of the servers and routers showed that we indeed were not connecting once outside the building. This was curious as we had taken the extra steps to make sure we had both a primary Internet service provider, and a secondary redundant but different company internet cable company. The Telecom big big boys, not some local provider either. Could they both be out?
Our ever vigilant tech crew had established a network of test access points from around the world, and sure enough, every one of those tests showed full access and connectivity to our eCommerce site. No problem in lost sales, except I am glad I did not have to be the one to try and explain that to our Founder / CEO.
As all good deeds eventually get punished, we discovered that on that day the highway crews were digging along the Interstate about 80 miles away from us, and their backhoe had pulled up both the major Telecom company cables buried right next to one another, about 50 feet from the pavement.
And of course there is the story of how I had to explain to (Different Company) our CEO the Hurricane flooded ATT center in New Jersey back in 1999 that had its switching center below grade in a flood plain. That was back in the days when all of the lower part of Manhattan and Newark NJ was off-line since all circuits went to that ATT facility.
Every day a new challenge.
