Data Centers

Sept. 11 teaches real lessons in disaster recovery and business continuity planning

The technology impact from the 9/11 disaster was enormous, with recovery costs in the billions. Business continuity and disaster recovery planners are well advised to learn from the experience and implement new strategies for the future.


By Dave Shore

This two-part series on disaster recovery and business continuity planning explores the technology and business ramifications following the tragic events of September 11, 2001. In this first installment, the author examines whether current plans were adequate to cope with the initial disaster as well as the effectiveness of longer-term business continuity plans.

The events of 9/11 at the World Trade Center have had a tremendous technology impact for the financial services firms in that location, as well as for those companies' global locations.

Approximately 8,000 Intel-based servers and approximately 5,000 UNIX servers were lost at an approximate replacement cost of $370 million. It has also been estimated that 30,000 securities positions (defined as trading, sales, research, and operations positions) were lost in the seven WTC buildings and another 15,000 to 20,000 positions in the adjacent buildings.

TowerGroup, a research and advisory firm, estimates that it will cost $3.2 billion to replace technology at the affected securities firms. Of this estimate, $1.7 billion will be spent on hardware—from trading stations, sales stations, workstations, PCs, servers, printers, mini-computers, storage devices, cabling, and communications hubs to routers and switches. The remaining $1.5 billion will cover services and software to install and connect the necessary networks, operating systems, and applications infrastructures.

TowerGroup's estimates are based on replacement costs of $52,000 each for the 16,000 trading desks (including turrets and multiple workstations outfitted with multiple flat-screen displays) and $5,000 each for the 34,000 general PC workstations (including monitors, extra memory, software, and networking equipment) lost as a result of the terrorist attacks.

A frequently issued statistic regarding disaster recovery is that one in five organizations that suffer a major incident never recover and end up going out of business. Last September’s events clearly illustrate why enterprises must plan for the unexpected and put processes in place to ensure that business continues in any possible scenario.

A two-tiered approach to disaster planning
Before September 11, anyone who had ever tried to persuade an organization to part with money to fund disaster recovery and/or business continuity requirements was quickly met with resistance. The difficulties in acquiring the necessary funding has typically led to a two-tier approach when coping with an immediate incident. The first part of the program, the disaster recovery plan, is put in motion when coping with an immediate (and often short-term) incident and traditionally provides for the most critical elements of the business—for example, typically no more than 20 percent of an organization’s workforce will have a desk at which to work. The second tier, the business continuity plan, comes into play during a longer-term, major incident, and would provide guidance on how to get the remaining 80 percent of the workforce back to work in the shortest possible time. This latter plan is now where much focus lies following the tragic events in New York City.

Sept. 11 created unique challenges for disaster recovery
All of the organizations affected by the 9/11 disaster have, to varying degrees, invoked disaster recovery plans. Many of the plans feature agreements with disaster recovery service providers that are contracted to provide emergency desk space.

Yet despite some prearrangements with providers, the events of Sept. 11 brought about some unique difficulties for all parties. Although a company may hold an individual contract with a service provider, the emergency space promised in the agreement is usually shared with other companies. Disaster recovery service providers base their plans on the probability that multiple clients will not invoke their agreements at the same time for the same space—an unforeseen issue in the case of the New York City events. While the client organization is busy restoring business as usual, the disaster recovery service provider is left with as many as six other clients who share that same disaster recovery space and cannot invoke that space if necessary. This leaves the disaster recovery service suppliers looking for alternate space as well, possibly competing with their clients to find suitable housing.

Further, service providers predict an expected duration of occupancy following an invocation. The typical expectation is that recovery from the incident will occur within days, or at worst, within two to three weeks—clearly not enough time for the organizations affected by the events of 9/11. Given the scale of the destruction in New York, organizations are left with the prospect of having to occupy a disaster recovery site for the full term of their contract, usually a maximum of six months. During this time, business continuity plans must come into effect and should include the acquisition of alternate premises (possibly a move to another location within the same organization) to allow the full complement of staff to resume normal work.

Plans must take all scenarios into account
The extreme scenarios that have occurred as a result of the terrorist attacks in New York City provide a good lesson in how sometimes, even the best-laid plans can fail. Organizations must now learn from these types of events and adapt current disaster recovery and business continuity plans to reflect these new issues and needs.

Dave Shore, a member of the Institute of Directors in the UK and president of Dave Shore Consulting, has worked with technology within the financial services sector since 1970. A former IT director, his work was recognized in 1993 with the IT For Business Excellence Award, cosponsored by The Sunday Times and Andersen Consulting. During 2000-2001, he was program director for a major global bank and was responsible for reviewing its UK disaster recovery operations and establishing a business continuity plan to cover its 3,500 UK-based users.


Stay tuned
In the second part of this series, we will investigate how the Internet can help disaster recovery and business continuity planners put forward more cost-effective solutions.

 

Editor's Picks