Solve this disaster scenario

First lets assume that the backup data center is set up so that it will continue to function no matter what happens to the primary data center. We also should assume that the data entry systems, with all the necessary comm lines, are duplicated so that people can get to the data if the primary data center is down.

Once those two assumptions are out of the way, the biggest flaw I see at first glance is the fact that the backup system may be too much like the primary system and that the half hour delay in switchover may mean that the disaster on the primary system is copied to the backup system.

In such a scenario, by the time you realize there is a problem on the primary system, you have duplicated the problem on the backup system. There are disadvantages in keeping things perfectly parallel.

The most innovative answers will be noted in the Disaster Recovery e-newsletter on March 25, not March 18. We apologize for this editorial error.

As a SME, a significant proportion of our DR plan is tape oriented. I noticed a distinct lack of any offline backup in your scenario. As you put it, "every byte of data is committed successfully to both the primary and DR data centres, with no exceptions", including I presume, deletions and viral infections. Without a secondary offline solution, there is little hope of rolling back any data based, non-physical disaster once it has been replicated to the DR site.

Actually, there are several fatal flaws.
My take on this is that the main problem may stem from the point of view of connections. It is all very well having a duplication of all data at another site, but do you have redundant methods of accessing that site? Also, in terms of the 30 minute failover, are staff in ALL remote sites trained and ready to make changes to router configurations, desktop settings etc to cope with the switchover?
How do you cope with a communications failure between thePrimary and DR centres? What happens if the line is the first thing to go and there are thousands of unreplicated transactions in the Primary site before the site fails completely?
Finally, has the solution been tested? My personal experience is that unless a full-featured simulation is done, even the finest planning may miss a small but vital feature.

To Whom It May Concern: IN MY OPION.Decrease fallover time to 20 minutes.What can be done in 30minutes can be done in 20minutes.Also if your team is in for the long run.they should have food supply,s ect, GEORGE ZAJICEK

First, It sounds like you are running full backups every day. At that rate those 4-TB's will go by-by real quick. You could run a full backup the first time and continue with incrementals after that to catch new data. Also what is the retention schedule for the backups? A month, 6 months, a year. The longer you are required to hold the data the more storage you need. The DR site should be making offsite copies of the backups to C.Y.A. in the event they go down and need to restore. 30 Minutes is way to long to find out if they have a real disaster. You can use an app. like "What's Up Gold" to keep an eye on your Servers/Network, so you would know quickly when you go down.
Just a few thoughts.
Thanks

I guess that I'm thinking more of a physical disaster than one involving data.
Is the backup location far enough away to prevent the disaster from wiping both off the map (earthquake, tornado, civil unrest)? What about staffing? Not just IT staff but key people from other departments. Do you have the means to shift people to the new location quickly?

There is no mention of network setup and connectivity to the end-users.

If I'm wiped out with the primary site, who will decide there is a disaster and notify the DR crew to jump into action?

When I talk about Disaster Recovery, I talk about actual Acts of God and Hardware failure.

Not Data Corruption. Data Corruption can always be dealt with at the backup level.

So my biggest problem with this scenario is the 1/2 hour failover time. Not to mention it was not stated that it was stated.

Fail-over means just that, it fails-over in real-time with little to no human interaction.

So to say that it can be back up in a 1/2 hour is not an acceptable down time. Actually that will probably kill most SLA's right there.

There are alot of ways to solve this problem. But it depends on how much cash you are willing to throw at it.

At the primary location, their should be 2 SANS's for automatic fail-over, with it's data being replicated to the backup disaster recovery site at regular intervals.

There should also be pvt lines going to the backup location plus an additional circuit over the public net over a different medium.

Same at the primary site. There mustbe multiple connectivity over different physical mediums.

It also states that there are servers "ready to be connected" to the secondary san. This means that they are not in a hot-standby fail-over mode. There must be some sort of human intervention there.

It also does not state how traffic will be directed to the backup location. DNS is notorious for caching entries. Have they set their ttl low enough to allow for expirations.

And where are their dns servers? And how are they set up? If you have your masters blown away, how will the slaves be updated with the new entries?

I'd say your chances of having at least one surviving data center are pretty good!

The fatal flaw is that your scenario does not address the user issues. Something took out your primary site; if your users survived, where do they go for the roofover their heads, the telephones, the PBX, the desks, chairs, PC's, etc.

If you do have a site from which users can resume work, how does it connect to the backup site?

All the hardware and software is in place, the technical staff can get the systems up within 30 minutes, thats fine, but there are no standby operation people running the show, they have to be physically shifted to the DR location, that to if the DRis in the same geographical area, it might just be possible or even nearly impossible to move the people within the same city in 30 minutes. Leaving aside if they are geographically apart say another city or even another country. Technically speaking the systems are up for operation, but physically no one available to operate it. This clearly means the DR planning also involves training some of the technical people the emergency operation task as required to run the show, otherwise the DR meansnothing but more downtime till the operative staff arrives.

True that the users have not been accounted for, so we assume that they have workstations and connectivity. Being a recovery planner, I will the technical issues of DNS propogation, data corruption, etc. to the techs. My concern is the full redundancy and switching the systems. Why would you spend the time and $ to keep a dev or QA server hot? Also, what are the critical systems that should be connected first and will 30 minutes meet their expected SLA to the business unit? Many of the other questions have already been voiced - Where is this data center? Is it on a separate power grid? Can we get the key people there in 30 minutes? Are the procedures documented so that other staff can bring up the systems if needed? Has the strategy been tested? More than likely, 30 minutes is not enough time to declare and get your team there during a major event.

True that the users have not been accounted for, so we assume that they have workstations and connectivity. Being a recovery planner, I will leave the technical issues of DNS propogation, data corruption, etc. to the techs. My concern is the "full redundancy" and switching the systems. Why would you spend the time and $ to keep a dev or QA server hot? Also, what are the critical systems that should be connected first and will 30 minutes meet their expected SLA to the business unit? Many of the other questions have already been voiced - Where is this data center? Is it on a separate power grid? Can we get the key people there in 30 minutes? Are the procedures documented so that other staff can bring up the systems if needed? Has the strategy been tested? Is there a plan? More than likely, 30 minutes is not enough time to declare and get your team there during a major event, even if you do have a Crisis Mgt Team.

There doesn't seem to be enough information presented in the scenario to make any justifiable conlclusions as to a fatal flaw. All posts so far are keying in on unknown factors: SLAs, automatic versus manual failover, end-users, connectivity to the DR site, distance between primary and DR sites, etc.

Perhaps the flaw in this scenario has to do with a technical issue. If so, I do not presume for it to be my job to know this. As a DR Manager, I rely on the technical SMEs in my organization tomake the right call when it comes to tactical technical solutions.

But, I have no reason to believe that the technical specifications and plans in this scenario have not been tested.

There is no fatal flaw presented. The existence of such a flaw will be only in its exclusion from the scenario. Which, er, makes this sort of a pointless exercise.

4 things were not mentioned:
-Availability of backup POWER supplies.
-YOU and a backup team?
-Has there been any TESTING of the plan?
-Is there any DOCUMENTATION?

1) In the event of power failure at the primary and backup site, is there abackup generator available?

2) The scenario was written for a generic 'you.' The word you can be singular or plural. Is there a team, or is there only one man w/ the authority to call the backup site? Has anyone else looked at the plan?

3) Have you ever tested the plan?

4) Is the plan documented? Is the documentation up to date?

Something that I see is missing is power. There is no mention of fail-over in power. What happens if a power station down the road goes out? You need two "trunk power supplies", plus stand-by generators to catch any intermittant power failures.
On top of this, what about a redundant communnication between the two sites? If you only have one ISP providing connectivity between the two and that ISP fails, all of your planning will be of no use.

Deviations from normal conditions are the causes of most incidents which in term can lead to an emergency. If these causes can be identified in advance, their potential consequences can be minimized by adopting certain measures to reduce incident?s overall impact on a site or the network. What are the measures to be taken and under what conditions? What is your primary source of information and the procedures to be followed incase of an emergency?

Do you have plan for unforeseen circumstances that may lead to an emergency?

.....

Lack of appropriate emergency planning, inefficient emergency management and implementation failure of the lessons learned from the responding emergency team are frequently the causes behind the unwarranted negative consequences made by those incidents.

There should be a plan identifying the individuals responsible for directing activities and providing the required level of support to the site(s) when dealing with a potential or declared emergency or disaster situation.

Your plan should also be intended to provide a quick reference of actions required, methods of implementation, schedule for updating both the plan, contact personnel and a monthly schedule to ensure both your sites and DRteam readiness to respond to all emergency situations at all times.

I'm not up to speed with DR, however in this scenario, I can imagine that corrupt data will be duplicated. And in that event, a worm, bug or whatever, will continue to replicate and or damage the operation of the system, and when on-line, continue to infect other systems until the backup also falls to the same fate as the primary system.

Does anyone know what the url is for the follow up article?