Discussion Locked

How often should you test a DR plan?

0 helpful
14 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

How often, if ever, does your organization test its disaster recovery plan? Do you think this amount of testing is sufficient? For best possible results, how often do you think companies should test DR plans? Share your comments about testing disaster recovery plans on a regular basis, as discussed in the Jan. 25 Disaster Recovery newsletter.

If you haven't subscribed to our free Disaster Recovery newsletter, sign up today! Click this link to subscribe automatically:
http://nl.com.com/MiniFormHandler?brand=techrepublic&list_id=e061

14 replies

by Member 30634329

Our policy is to have a formal documented test of our DR plan once a year. Unless the plan itself changes before the next formal test.

by black panther

Sometimes the testing itself can produce more problems ( ie if it ain't broke then don't fix it)

It's all about risk - if you are 24 x 7 then you wouldn't have much time to test at all.

by stress junkie

The most appropriate answer to this issue must first match the size and the nature of the business. A fifty person business, or smaller, may have one IT person. A business with 50,000 employees may have thousands of IT employees. A financial institution absolutely should practice their disaster recovery monthly. A car repair shop could probably practice their disaster recovery once a year.

I believe that disaster recovery should be practiced frequently. Everyone in IT support should know exactly what they would do when the disaster happens. Failing to do this will result in people standing around with their thumbs up their asses saying 'What should I do?' if a disaster happens.

The fire drills can be managed in such a way as to minimize the impact on daily activity. For instance you don't have to shut down the entire IT department at one time. You can have individuals practice their part of the disaster recovery while the remaining people continue with their daily work. This would have the benefit of more closely immitating real life disaster recovery. Each person should be able to do their part without supervision when the time comes. If a disaster actually happens some people will arrive sooner than others and some people will be on vacation or otherwise unavailable to help. If each person is used to working on recovering from a disaster independently then they can begin their part as soon as they are available when the real thing happens.

My response to the person who suggested a fire drill of once a year is that people cannot be fully prepared to perform their part of disaster recovery unless they practice the procedure more frequently than once per year. I think that monthly fire drills ensure that people are fully trained at all times and that those people will be confident enough to make good decisions quickly when required.

The person that suggests that 'if it ain't broke then don't fix it' is taking the least responsible and least sensible approach imaginable. How can you know if something is or isn't going to work unless you test it? Having some pretty document describing a disaster recovery plan isn't enough. There may be practical problems with the plan that you won't know about unless you test it.

One of the most important characteristics of system administrators is responsibility. It isn't enough to think that you probably know what to do when a disaster happens and that you can just make up a recovery procedure as you go. You have to know every step that you are going to take before the disaster happens. You have to know the phone numbers that you are going to call and the people that you are going to speak to when the time comes. You have to know when you can expect to receive your off site backups. You have to know where your alternate site equipment is located and who to speak to in order to gain access to it. The only thing that you can be sure of is that if disaster happens you will be confronted with dozens of issues that you couldn't foresee unless you practice your procedure.

While the car repair shop won't have an alternate site I believe that all businesses of any size can afford to maintain off site backups. Any business can afford to rent a safe deposit box at a local bank. That approach would work very well for 99% of businesses.

In conclusion I believe that disaster recovery has to be as familiar to IT support personnel as are their passwords. There is no room for playing it by ear. That is irresponsible. Failing to practice existing disaster recovery procedures is also irresponsible. Disaster recovery plans have to be scaled to the size and nature of the business, but all businesses are capable of keeping off site backups of their business data.

by agbons

The choice of testing a DR depends largely on the type of data - whether it is critical to company's operation or not. For mission critical data, I suggest a weekly test.

My organisation is into drilling oil/gas well. Part of my DR plan entails a weekly backup to disk in central server which implements disk mirror. From time to time I keep a copy of the data/file in CD rw. This gives me a sense of security somehow!

A good IT pro can tell if his/her DR plan will pass any time.

by bfilmfan

You should have an annual review of your total DR plan, with tests of specific portions monthly.

by stress junkie

Your response presents a compelling argument for your opinion. You have clearly given this issue a lot of thought. The logic of your arguments supporting your position is flawless. I would only suggest that you reduce the verbage a bit. Lengthy answers can be tiring to read. Otherwise your wisdom and expertise are clearly illustrated by your answer.

by house

I would only go so far as to test the validity and quality of your backups and disaster recovery solutions. I wouldn't run full scale tests at all when we are talking about live data. Running labs and checking the validity of the real backups in a mock setting is good enough for me. I don't even want to think about [i]live testing with live data.[/i]

by stress junkie

Disaster recovery applies to the kinds of things that destroy your primary equipment and facilities. Whether the disaster is a fire in the computer room or a tsunami that destroys the entire building the disaster recovery procedure would by definition be using equipment that is assumed to be spare. I don't know where you got the idea that disaster recovery procedures would involve messing with the primary production servers. The disaster presumes that the primary business equipment has been destroyed, so practicing disaster recovery would necessarily involve backup equipment, an alternate site, and whatever other redundant resources have been made available for this purpose.

by house

Disaster Recovery consists of many things, and it is not just reserved for physical damage. When people are talking about live data, I think of ripping out a blade and swapping it for a new one, rolling back a corrupt database, implimenting backup hardware in a live setting, the list goes on.

One of the strategies that were mentioned above, is live testing with live data.

PS - DR does not necessarily mean that you have servers ready to take over the task. Although this kind of redundancy is common, it really depends on your network.

by dafe2

Components are tested quarterly. A full DR exercise is performed once a year. All players and their backups are present....and roles played out & documented.

by msmills

Wow I'm glad I don't work for you guys - you seem to have backup and DR confused. DR is about business contiuation not lab rat backup restores.

The test comes whenever - employees are told to report to an alternate site - and if they can't sit down and work when they get there the test is a failure. The test is best run when least expected - I've never had much luck with scheduled disasters.

If the company you work for thinks DR is restoring the last backup tape to another box I'd either be looking for a new job or convincing them there is a little more to the subject. Either way I would get those resumes out because if there is a disaster odds are you won't be employed.

by ksat

You are absolutely correct. Many of the responses I saw here involve normal operations and maintenance procedures - not DR!

There are 2 items here: DR - disaster recovery - involves the ability of a company to respond to a catastrophic event and how long that company can go without it's financially significant data (RPO) and how long it will take that company to get things up andd running as it was just prior to the event (RPO). Strictly speaking, this is only about applications and it's associated data.

The other side of things is Business Continuity. This refers to the ability to withstand any disaster and continue operations as 'normal.' This, technically, has nothing to do with IT and more to do with business processes and workflows.

So, for those of you thinking that repairing a CEO's laptop is DR or that swapping a RAID drive, then I believe you need to do a bit more research.

Regarding testing schedules: Ideally, full-scale tests should be conducted annually, at minimum. Smaller, more application-centric testing can occur (and should occur) weekly or monthly. Obviously, this will be greatly dependent on your specific environment and organization size.

by dafe2

The original post is not very clear (IMO).... When (I) think DR, I'm more inclined to think as you say - It could include anything from a CEO's laptop going ti.ts up, a dead array on a server or a failed switch even an obscure PC that no longer boots. Not (necessarily) a huge impact.

Now, Business continuity does include an IT COMPONENT which in many cases may include a cold or warm site (with dark fiber) or a hot site miles away.

IMO, there's a BIG diference between DR & Business continuity testing, so nice catch.

This discussion is closed to new replies.