IT Employment

Testing your plan every year (or at least major parts of it) is probably the only way to find the big holes.

In our 2003 test, we went to test one system and found it didn't work at all. Reason: we didn't restore the middle tier server!

A little research showed why: the support people just forgot about it, and more importantly, the "dashboard" system that indicates which systems are up/down/slow didn't even list it!

Furthermore, the service level agreement for that system didn't list it either!

We also found that a failover server for another system would have failed over to 8-month old data because the dialy script that copies changes from the production system to the failover system hadn't worked in ages.

Most interestingly, I also noticed that the MS SQL server that one my admins created from scratch for one test had NO patches on it to prevent a Blaster infection (SQL SP3).

So testing separates the sheep from the goats and gives you good insight into your procedures. Everyone hates doing recovery testing, but a lot of CYA work is done quickly after holes are revealed.

I meant SQL Slammer, not blaster.

Semantics, semantics. When we view DR simulations as actual "tests", we promote the idea that a success negates the need for further testing. This is obviously not true because DR simulations are more than verification of functionality. They are events that provide the opportunity to hone skills, to practice old and new techniques, and to bring new blood into the process of recovering the organization.

Testing indicates success or failure in many corporate cultures. Yet, as BC/DR professionals, I'm sure many of us understand that finding failures in a simulation is the best time to find them. So, a failure in turn becomes a success if you are able to isolate the root cause and implement corrective measures.

Just as an athlete exercises to maintain top performance ability, DR plans need exercising to ensure they can perform to the level of expectations during a time of crisis.

DR simulations are also a very valid conduit for gathering requirements. When something in the organization changes, especially on the business side where the information may not trickle over to the DR group in a timely manner, a DR simulation exposes this new requirement when the business groups make it known that they can't perform some of their job functions without x, y, or z.

I suppose what I am trying to get across is that as an industry we need to get away from labeling the time spent simulating recovery of systems, applications, processes, and functions as "tests" and promote the idea that these activities are an ongoing part of the larger program that is disaster recovery and continuity management.

The final step in the DR Plan testing/exercise process is the postmortem meeting and management report. This is often overlooked as an important part of testing. A Management report should document all of the test objectives that were and were not met, as well as the challenges and issues that came up during the test. All of these issues, in turn, should be documented in a project task list to ensure they are addressed in the DR plan in the form of updates and/or changes.
Not only does this report serve to inform management of the state of their DR planning process, but also aids in future audits by internal, external and regulatory auditors by showing what was accomplished, and that changes were made to the DR plan based on the test's findings. The postmortem meeting is to finalize any thoughts about the test by the participants, which should also aid in the planning of future tests.