By Mike Talon
Even the best disaster recovery plans can easily fall prey to failure. Your plan may encounter errant service packs and software updates, hacker infiltration, and a host of other unexpected situations.
But the number one factor that can decimate your DR plan—without any help from an actual disaster (although it often causes them)—is an internal situation. It's the presence of bad change request procedures in your data centers.
Change request procedures generally refer to the concept that admins must first obtain multiple levels of authorization before they perform any time maintenance or upgrades on server systems. This process serves two purposes:
In a best-case scenario, an organization has defined a clear procedure for requesting and obtaining the necessary authorizations, scheduling the update, and performing the work. This allows admins to make multiple changes during a single maintenance window, and it keeps everyone up to date on the builds on the various machines.
In a worst-case scenario, this process wreaks havoc across an enterprise.
Redundant data systems, replication schemes, backup systems, and other DR solutions are not the answer to bad change request procedures. If nothing else, poor change request procedures ensure that the production systems and the DR systems don't match, which generally leads to failure on failover. But keep in mind that if you're not tracking change requests, DR systems can easily miss critical patches and software updates, making them security hazards on top of invalid failover resources.
Establishing change request procedures is something that almost any company can do with little aggravation. The biggest hassle is defining who requires notification for what and ensuring that anyone who will make changes knows this information.
Of course, smaller shops have the advantage. Only a few people may need to be involved in the entire process, which makes it an easy task to set up and notify those who need to know the plan.
Larger organizations need to draft a written document, and they must ensure that any employee who could conceivably make changes to the servers is fully aware of the procedures. They also must ensure that those who shouldn't be making changes do not, or else the entire change request procedure does no good.
No DR plan can overcome bad change request procedures, and it can actually create more of a hazard if the organization ignores such procedures. Make sure you've fully fleshed out your change request procedures—and that employees understand them—to guarantee that you don't accidentally create a disaster when attempting to fix something on your primary systems.
Mike Talon is an IT consultant and freelance journalist who has worked for both traditional businesses and dot-com startups.