I've been participating in Disaster Recovery exercises at my company for close to 7 years. We perform actual exercises twice per year and we learn of a new challenge or we learn something new at each exercise.
A couple of bullet points here, I'll be as brief right now because I don't have a lot of time at the moment, but I would enjoy a discussion on this topic with others.
* DR/BCP (Business Contuation Plan) - Probably the most important piece to DR/BCP planning - have a well documented plan.
* Practice - Practice your DR/BCP plan on a regular basis. There are so many pitfalls in a DR scenario that you likely won't think about or discover unless you actually go through the motions.
* Documentation - Your technology must be documented so that what you want to rebuild at DR can be built off of the documentation by someone who doesn't know you, your company, or your product. It's a morbid thought but in your DR plan you or your co-workers should anticipate death meaning someone else who doesn't know your environment as well as you has to rebuild it at the DR hotsite. Don't assume the person rebuilding your environment has as much technical knowledge as you. That's why documenting EVERY SINGLE STEP in a fool proof manner is important.
* Update documentation - Ok so now you're ahead of the game because you've got all your systems documented. Systems are dynamic and change in time. It's important to remember to KEEP your documentation UPDATED as your systems change. This one is easy to forget.
* Hotsite - Your plan should include a designated "hotsite". This would be a building, a warehouse, office, whatever, where you can occupy the area at a moments notice, and all necessary equipment (phones, servers, fax machines, telco, routers, etc.) are waiting in standby mode for your use. Sungard (formerly Comdisco) is one such company that handles facilities like these at several sites across the nation. Your hotsite should be nowhere near your primary site which you are recovering because a catastrophic disaster of your primary site would likely wipe out the hotsite as well.
* Critical systems - All of your systems should have a priority assignment which dictates the order in which it is restored at DR. Bread and butter application servers would be examples of things that would be restored first. You can't restore everything at once and furthermore you will not be able to restore "everything". Concentrate only on the critical components of your business.
* Validate backup tapes - Hopefully you are backing up to tape on a regular basis. An additional step to be performed is to validate the integrity of your tapes. We've gone through so many problems at hotsites with tapes it's unbelievable.
* Hardware compatibility - Make sure the hardware you have at the ready at your hotsite will be compatible with your applications. Consider CPU, Memory, Operating system, etc. Also consider tape backup hardware. A 4mm DAT drive at the hotsite does you no good if you will be bringing DLT tapes onsite if and when you declare a disaster. An additional note on backup hardware and compatibility: Standardize on a backup vendor, hardware vendor, and a compression method for your backups. We've found that intermixing compression types with different drive vendors and different backup software vendors simply doesn't work at DR. If you don't have the software or hardware at you DR site that will read the compression applied to your tapes, your tapes will be unreadable. Because of this we have standardized on software compression with one and only one version of Veritas Backup Exec. If you have many tapes to restore at DR, consider allocating many standalone tape drives at the hotsite so you can have many restores going simultaneously as opposed to babysitting 15 tapes sequentially.
* Account for employee fatigue at DR. The recovery process itself is an around-the-clock process (so should your practice exercises). Plan to rotate shifts for your employees responsible for recovery. Book hotel rooms at or near the hotsite for employee rest. When fatigue sets in, efficiency drops and mistakes start happening.
* Validate your exercises - Once you get into the habit of practicing your DR recovery process, it's important to have your business units come in afterwards to "use" the system. This is the ultimate test. Document what works and what doesn't work. If WinZip 9.0 won't load for Sally User, figure out where the mistakes were made and update your documentation to improve for the next time.
* DR Container - dedicated a steel box type container that will hold all of your DR documentation plans and necessary software. Store this container offsite (at someone's house, or at an offsite vault where you might store backup tapes). In the event of a disaster, your container and all its DR plans are safe. You simply pick up the container and head to your DR site. EVERYTHING YOU NEED TO REBUILD YOUR WORLD SHOULD BE IN THIS CONTAINER. At your practice exercise if you find that you forgot to put the Windows 2000 Server CD into the container, you have failed.
* Last one before I need to run here - Learn how virtual computing can improve the recovery time and ease for your environment. I highly recommend investing in a VMware product or Microsoft Virtual Server. Entire computers and servers can be burned to a CD or DVD and made portable to bring to your DR hotsite.
Jason Boche
Minneapolis, MN