DR documentation and distribution

You don?t know the misery until you actually live it. Thank you Mike for your correct approach that should people take when DR occur. Keep up the good work and look forward for more articles.

Importance of DR documentation has been nicely brought out. The scenario when CFO could be the only person to bring up the e-mail server is well narrated.

We have done repeated DR testing of our sites and these drills have amply shown the importance of the documentation as elaborated by Mike.

In fact , there is yet another aspect that we found to be extremely crucial. It is the availability of all information in synch across all locations whether it is primary site or DR site (for ready reference). We have resorted to hosting these information at a different location on the web with an emergency management system/framework implemented to provide a framework. Such information include emergency contact lists and phone numbers- updated all the time,details that would be needed for recovery team (Checklists,Task lists, alerts and messages). Such a system has helped in ensuring that current information is available to all participants and also is accessible when the sites could be down.

I've also seen systems with perfect as-builts as of the go-live date, but nobody required to do routine updating on the documentation. During the cutover and warranty period, settings were changed and configurations tweaked. Since nobody updated the as-builts with these changes, no system description after 30-60 days. Give it a few more months of updates, upgrades, enhancements and your operating system is miles away from the as-builts. Changes were made by a dozen (or more) support people over time, so nobody knows the entire current configuration.

Now you rebuild this system!
Moral of the story, change/upgrade documention is critical. Out of date materials are often worse than none at all, because they could restore fatal errorsthat had been corrected. In some database and ERP systems, the current data won't even reload properly into the original as built configuration.

If EVERYONE on the IT staff isn't trained on this, if they don't have access to the backup equipment, then you will be explaining to the boss the next day.

We are going so far as to color code and take digital pictures of our racks for documentation in a manual notebook kept right beside the power out light and flashlights in our server room.

We have laptops configured with fax modems and extra copies of notification forms in order to notify remote sites in case power is lost.

If you believe but can't imagine, have a saturday drill when you actually throw a breaker, pull a router, or randomly power down one a server.

Bet you 10 bucks the company fired the guy that originally put the system together because he probably wanted a raise. So instead of giving it to him they fired him and hired in some young paper MCSE at half his pay.

He probably intentionallyleft that black box out of the DR plan as a "gotcha" just in case he ever got fired and now he has read this story and is laughing his butt off somewhere! Never let someone else do your work for you.

We used to use a couple of cd towers (Todd and ONELINE) that worked on our Novell 3.x and 4.x system (actually were "novell servers"). When we moved to Novell 5.x, we could no longer control the towers' configuration and had to retire them. Ever since, I've been looking for a cd tower that could hold more than 20 cd's AND had a NIC, not USB. Our techs need those cd's from any number of client pc's and don't have time to go back to one specific pc for the drivers/files. We don't like the ideaof taking up so much server hard drive space. Any suggestions?

Also, we see media management as a security issue as much as a logistics issue. We paid for only so many licenses and media. We make a copy of the original media and lock it away. Techs have access to the copy when they need it.