CXO

Lock IT Down: Live by the Boy Scout motto

Last August, in a massive blackout, 80% of New York State lost power. Not a single customer of Con Edison Communications, however, lost power. Here's why.


At the height of the massive blackout that rolled across the northeast last August, CBS News reported that 80% of New York State lost power. Yet, incredibly, at the onset, not a single customer of Con Edison Communication, the subsidiary of Consolidated Edison Company that provides data and voice transport products and services to metro New York City, lost service. Joe Laezza, Vice President of Network Operation for the New York City-based company, credits his firm's successful business continuity to its rigorous disaster recovery preparations. "The time we spent establishing our disaster recovery plan and running drills to test its viability really paid off when it came time to respond to a live widescale incident like we had during the summer of 2003," he said.

The real dividend stemmed from the section of the document that outlined the responsibilities of every participant in the emergency management team. "Within the plan," explained Laezza, "we defined who the emergency management resources are within Con Ed Communication and what each of their roles and responsibilities are for responding to a crisis like the power outage." Despite the monumental difficulty in traveling around Manhattan at the time, within the first hour and a half, the entire emergency management team was assembled, the crisis bridge activated, all the assignments divided out, and the response underway.

Taking a page from the Boy Scouts of America handbook, Con Ed Communications' strategy can be summed up in two words: Be prepared. Laezza offers TechRepublic readers some tactical advice on just how to do that:
  1. Divide and conquer. Whether you're a network service provider like Con Ed Communications or a user of network services, you need to do a risk assessment on two levels: 1) the impact of possible disasters on your internal operations, and 2) their impact on your ability to provide service to your customers.
  2. Respond and restore. Your first task should be to outline interim solutions to get yourself back in business. Your second task should be to outline strategies for restoring operations back to normal business practices. Separate your emergency management resources into two parts as well—a response team to handle the immediate crisis and a restoration team to get you back to business as usual. Make sure that you establish specific methods and procedures for each team to use in the event of a disaster.
  3. Review and challenge. It's not enough to review the vulnerabilities of your own operation. You need to make sure that your carrier and your business partners have viable recovery plans in place, too. Your telecommunications and power providers are essential elements of your operation. Without their services, you'd be hard-pressed to maintain continuity. What about your property manager? Does the company have a clear set of procedures to get the physical plant back in usable condition quickly? What about your software support provider? The operator of your data warehouse facility?
  4. Revisit and revise. Make sure that your disaster recovery plan stays a living document. Revisit the content on a regular basis so that, as your business evolves, your plan can be updated accordingly. Something as simple as a change in the management echelon can undermine a plan's viability if the emergency management team list isn't updated to reflect new players.

After 9/11, facing worst-case scenarios no longer seems outside the realm of possibility. Laezza suggests that your risk assessment checklist should include hard questions like: If an office were to totally disappear, would you be able to recover and restore business as usual? To answer that question, you need to examine who resides in that location. What work do those employees do for the business? What is the critical nature of that work? Can you potentially recover that work in a different location? The answers may lead you to devise a backup and redundancy strategy to cover your bases.

On a less extreme note: If you were unable to get into a physical plant—due to vandalism, flooding, or even a faulty air-conditioning system spewing out noxious fumes—would you be able to recover and restore business as usual? In such cases, Laezza recommends using technology to your advantage. Voice over IP, for instance, can enable you to recover communications in different locations. Consolidating the traditional phone network with your data network would enable your employees to work from home or other locations even if the office building was quarantined.

As an expert in network operations, Laezza offers some additional advice to maintain high availability of your corporate network during natural or man-made disasters:
  1. Don't put all your network eggs in one basket. Diversity will ensure uninterrupted availability of your business. Make sure your third-party provider's network doesn't have a single point of failure. It should encompass dual fiber optic routes and network hubs that support your network in separate physical locations. To improve your chances of continuity, you may also want to consider using separate carriers to support core operations.
  2. Seek out carriers with self-healing networks. Not all carriers deploy ring configurations that can withstand fiber cuts or outside plant explosions without losing service. Make sure your carrier can withstand the loss of a node within the ring itself and not lose service. With self-healing networks, fiber paths support three terminals in a ring. If you lose a path because of a fiber cut— for example, if a street repair crew gets a little careless—the traffic actually switches to the other side of the ring, and service remains uninterrupted.
  3. Consider turning your provider network over to carriers who live and breathe network maintenance and security. In recent years, some enterprises have taken on the role of operating their own private networks. But the approach has some serious pitfalls. Businesses commonly don't have the specialized talent to support and maintain a full network. Without the training, manpower, and preparation time necessary to cover all the bases, your exposure in the event of a disaster could be quite high. Outsourcing your network operations to experts in the technology will inevitably reduce your operations costs and liability in the case of a disaster.

While all of these tips won't guarantee that your business remains operational during severe conditions, Laezza says they do provide a useful framework around which you can prepare for the worst.

Editor's Picks