Data Centers

Disaster recovery worst practices: Don't look at your logs

Consultant Peter Herbener has compiled a series of disaster recovery worst practices based on his experience. Implementing a state-of-the-art backup system won't help if you're not paying attention to the logs. If you're struggling with the basics, here are some tips to get back on track.

Most backup software gives you a way to automate the process. Depending on the amount and complexity of your organization's data, backups can be a time-consuming process. Who wants to sit and stare at a computer screen for an hour or more while it just makes a whirring noise?

Automating your backups has other obvious benefits, such as running them during off hours, when few if any users are working. Speed and performance issues are not as important, plus the backup doesn't have to handle files that are being changed. Even if you're doing a daily backup, it seems like all you have to do is pop yesterday's tape out and put today's tape in. Also, you can do this at any time before you leave for the day.

DR worst practices

Don't miss the rest of the series:

Download the complete disaster recovery worst practices series in PDF format.

Nightmare: There's nothing on my backup tape!

I knew a system administrator who had a state-of-the-art setup. He had top quality software that could back up multiple servers to one location. He had a tape changer so that the software could automatically swap tapes — either daily, so he didn't have to change tapes ever day, or in the middle of the night, if backups ran more than one tape.

It took a few weeks for him and another IT staff member to get it all installed and running, but finally it ran smoothly. This was complicated and expensive, but it was worth it because the company had nearly 100 employees and handled many high-dollar transactions.

Then one day, that same IT staff member got a job at another company. As part of the routine, the sysadmin deactivated the ex-employee's user ID to prevent him from being able to access any of the servers in the future.

After about one month, the sysadmin needed to look at some files from a backup for research purposes. Fortunately, he didn't have to recover from a true disaster — because there was nothing on the tapes. It turned out that the ex-employee had given the backup software his user ID and password for the software to access files from the servers. When the boss deactivated that user ID, the backup software could no longer access any of the servers, so it couldn't back up any of their files.

Had there been an actual disaster, the company would have completely lost months of data. That could have cost the company many thousands of dollars.

Now, there are plenty of mistakes here. For one thing, it's best if your tape backup software doesn't need a normal user ID and password. If it absolutely requires a user ID and password, it's better to create a special one that isn't normally used by anyone.

The worst mistake, though, was that nobody bothered checking to see if everything was still working okay after the initial implementation. When someone finally got around to looking at the backup logs, error messages clearly showed that the software was unable to log in and that it hadn't backed up any files.

Best practices for monitoring your logs

We like to think that when we automate something, we can just set it up and leave it alone. However, it's very important to monitor automated processes to see if errors are occurring.

Any halfway decent backup software will try to tell you if something is going wrong, usually in its log. The log may be a simple text file, or the backup software will include a menu selection to allow you to view the log; it might even be set up to e-mail the log to you. You just have to know what to look for and take the trouble to pay attention.

The log might be a "summary" or "detail" log. A summary log should at least show you what time the backup started, the number of files backed up, how much space it took, and the time the backup finished. A detailed log might list each file. Usually, a summary is all you need.

How to monitor your backup logs

Look for error and warning messages.

This is the obvious first thing to watch out for. Take all error messages seriously. Have you ever heard anyone say, "Don't worry about that — I don't know what it means, but it always gives us that error"? That can be a dangerous attitude.

If you don't understand what an error message means or what caused it, you have no business ignoring it. This gets you in the habit of ignoring error messages, in general. Ideally, you should try to fix and problems reported by an error message.

Check to see if anything obvious is missing.

Did it back up all of what you expected it to back up? Is there a server, volume, or disk drive missing?

Make sure the numbers add up.

Does the number of files backed up look right to you? How about the total size of the data? The numbers will never be exactly the same, but if they go up or down dramatically, you should get suspicious.

If they go way down, maybe the backup skipped a lot of files, didn't have proper permissions, a server or drive was turned off or disconnected, etc. Even worse, could a large chunk of files have been deleted by accident or by mischief?

If the volumes go way up, you should try to find out the cause as well. Was a new application or database installed? Is someone sharing her music collection with the department? Or are you somehow backing up the same thing twice in one night, and perhaps completely skipping something else?

Check the time.

Did your backup start and end on time? Did the total elapsed time take approximately as long as you would expect? This is another way to get clues that you backed up much more or much less than usual. It can give you a heads-up that there might be some other performance problem that you need to address.

TechRepublic's free Server NetNote newsletter provides valuable insight into deploying, planning, and purchasing hardware for your data centers and server room. Automatically sign up today!

Editor's Picks

Free Newsletters, In your Inbox