Data Centers

Preparing for large-scale disasters and anticipating recovery time

There are some disasters that challenge any business contingency plan that you might put in place. Make sure you set realistic expectations for recovery time in the event of large-scale disasters.

I usually plan my topics several weeks in advance, and it so happens that I chose to write about large-scale disasters well before the attacks in London. It is with some trepidation that I write on this topic in the wake of such recent tragedy. To the families and friends of all those touched by this attack, my deepest sympathies and, in the spirit of carrying on that the British have modeled for the rest of the world, I dedicate this column to the memory of those lost on July 7, 2005.

I've written on the subject of large-scale disasters before, and it is never an easy subject to cover. There will be disasters that cannot be planned for, that cannot be protected against, and which cannot be recovered from in the way we are familiar with when discussing disaster recovery (DR). Many of these types of disasters are natural, with tsunamis, hurricanes, floods, and fires as common examples. Others, as we were so recently reminded, are man-made, but any disaster on this scale cannot be simply recovered from by restoring from backup tapes. So how do you bring back the business when such severe disasters strike?

Set expectations for worst-case scenarios

Believe it or not, many executives will continue to cling to the misguided belief that technology has an answer for everything. The sheer number of non-technical staff who believe this is second only to those who believe that DR doesn't cost anything to be done properly. Many executives simply don't understand that not everything can be planned for, and not every calamity can be recovered from immediately. You must prepare them for the possibility that a large-scale disaster could disrupt your data systems for a period of time, or even take them offline permanently. If a fire destroys the building, your servers are going with it, and no amount of backup tapes are going to be able to restore the data until a new location is found, and new servers are set up.

Also, keep in mind that your DR plan for data systems will probably take second place to the fact that buildings are destroyed or inaccessible, phones and data connections are not functioning, and that people may not be able to reach their workplace because the area is inaccessible. This means that you may be unable to reach backup systems, vendors cannot supply new equipment, and staff cannot start a recovery process, even if you have everything else in order. The severity of some of these situations can be mitigated by keeping backups off-site or replicating to far-flung facilities, but even so, if the people who know how to recover the systems are unable to get on the right flights or their cable modems and DSL lines are not functioning for remote access, you're not coming back up any time soon.

Speaking of the human effects of these tragedies, you must keep in mind that these disasters often cause loss of life, and some of those lost may be key employees in your firm. For the survivors, there is the psychological impact of witnessing the events, and dealing with the aftermath. You cannot expect someone who's just survived a massive earthquake to be in an emotional state to immediately begin restoration efforts. This will add to your timeline for eventual restoration of systems, and must be taken into account in any business continuity plan that has a chance of success.

As you can see, in cases of large-scale disasters, DR planning moves from the realm of fast recovery to the realm of eventual restoration of services. With time, new locations can be found, or existing locations can become accessible again. New hardware can be procured to repair or replace that which was lost. Employees can return to work and begin the process of healing. All of this, however, is a matter of time. In preparing for large-scale disasters, the most important thing is to remind senior management of the limitations of even the best contingency planning, and that recovery time can stretch into days or weeks, if not longer.

People have shown an amazing capacity for recovering from every calamity that impacts us. The technology that we create is part of that ability to bounce back, and we must be willing to give it the time needed to react and recover. When massive disasters strike, keeping both the technical and human factors involved in the proper perspective will allow you to restore your business and the services you provide to your community.

How well can your organization deal with an emergency? Automatically sign up for our free Disaster Recovery newsletter, delivered each Tuesday, and make sure you're prepared for the next catastrophe.

Editor's Picks

Free Newsletters, In your Inbox