Disaster Recovery optimize

10 things you should know about automated backup programs

Automated backup programs greatly simplify administrative tasks. However, it's possible to become overconfident in an automated backup. IT consultant Erik Eckel reviews 10 things about automated backup programs that could save you and your organization from a recovery nightmare.

This article is also available as a TechRepublic download.

Automated backup programs, whether used to create local backups or copy data offsite via high-speed Internet connections, greatly simplify administrative tasks. Properly configured, automated backups -- including Remote Data Backups, Spare Backup, Dr. Backup, Yosemite Backup, Windows NT Backup, and Symantec/Veritas Backup Exec -- not only ease an administrator's workload but provide some peace of mind.

Eliminating the daily pressure of having to manually back up an organization's critical data opens valuable time that can be dedicated to other responsibilities. However, it's possible to become overconfident in an automated backup.

Alaska officials, for example, recently revealed that a computer technician accidentally deleted data on a hard drive. Seemingly no trouble, the case took a bad turn when, attempting to recover the data from a backup tape, the state found the media unreadable. Recovery costs are estimated to exceed $200,000.

Review the following 10 things to know about automated backup programs. They could save you and your organization from a similar nightmare.

#1 Tapes aren't trustworthy

It's a sad truth. Many expensive tape backup systems fail when needed most. What's worse, many tape failures are never caught. Whether it's a case of a tape drive requiring cleaning or media failing over time, often tape errors aren't caught until too late. Just ask Alaska's Department of Revenue, whose $38 million oil account (including 800,000 electronic images) had to be painstakingly rebuilt by more than 75 employees because backup tapes proved unreadable.

#2 Tape maintenance is dicey

In addition to tape drives and tapes themselves proving questionable, even proper-operating media are only as good as the operator. Unless administrators and others charged with rotating the actual tapes complete the task on time using the correct media, tape backups can prove worthless. Even veteran IT professionals occasionally insert the wrong day's tape or confuse recovery sets. For this reason, it's important that schedules and media are carefully monitored and tracked.

#3 Data locations change

Data locations move and change over time. For example, an organization's public relations files might originally be installed within a server data folder labeled PR. Following an acquisition, a new storage strategy might be implemented in which those same PR documents become part of a Marketing folder. The same thing happens with databases, e-mail accounts, user directories, departmental archives, and other data. Unless backup operations are updated every time data storage locations change, backups run the risk of missing critical data.

#4 Backup operations occasionally fail

Just because a backup operation is scheduled does not mean that backup procedure will complete. Electrical outages occur. Thunderstorms intervene. Backup media fills. Backup drives get dirty. Systems freeze. The list of elements that could derail a backup is unending. Thus, you should never consider backups covered just because they've been scheduled. Instead, make reviewing backup logs a daily routine. Better yet, make restoring backups to test their efficacy a regular event.

#5 Backups back up bad data, too

When backup operations complete properly, they tend to complete exactly as programmed. Backups don't care if whole directories or partitions have been deleted since the last time they ran; backups usually back up what they're told to back up. For this reason, administrators should not depend upon a single backup set. Users occasionally delete whole folders and directories by mistake but sometimes require several days to realize the error. If your organization is working with only a single backup set updated daily, the likelihood of recovering the erroneously deleted data decreases every day. Maintaining multiple backup sets (or performing differential backups throughout the week) provides organizations with additional options for recovering data.

#6 Databases and Exchange require TLC

Many applications -- including those that depend on Microsoft SQL Server and the Microsoft SQL Server Desktop Engine (MSDE) to power their data -- store their most critical information within multiple database files. Unless the complex instructions that link the information between those databases in meaningful ways is also backed up, just having those database files saved to a backup drive won't enable successful restoration. Be sure to follow the manufacturer's backup guidelines when working with such third-party software.

Exchange servers need special treatment, too. E-mail servers require applications that can perform online backups, as it's impractical to assume an organization could down e-mail servers during specific windows daily just to complete backup operations. Instead, organizations must ensure their backup applications support online or active operations. In the case of Microsoft's popular e-mail server, such programs are described as being Exchange-aware.

#7 Some apps work better than others

Many vendor promises amount to sweet nothings; not all products work as promised. Some applications fail to back up all the files, folders, and drives you specify. Others perform a differential backup even though you called for an incremental. Still others fail to properly write data to specific media or don't complete within reasonable timeframes.

Worse, competition within the online backup space results in many providers going out of business. Often firms go under with little notice and take your data with them. So shop carefully when considering software manufacturers and online providers. Reputation and reliability typically outweigh cost savings when selecting a backup partner. Whenever possible, don't forget it's a best practice to first test an application before deploying within a production environment, too. Doing so helps reveal anomalies and incompatibilities before damage can be done.

#8 Documentation is critical

The best defense against data loss, and a crucial component of any disaster recovery plan, is documentation. Only by documenting which systems are backing up what data and when (and where that data is stored) can an organization have confidence its critical data is properly protected. In addition to tracking this information, documentation should provide instructions for testing backups to ensure the backup sets enable proper recovery.

#9 Proper backup strategies require regular reviews

Data locations change. Often, documentation doesn't keep pace. As a result, it's easy for an organization's backups to begin tracking the wrong data. IT departments can help prevent disaster by scheduling regular reviews of its backup strategy. Scheduling quarterly meetings to review backup strategies can help ensure backup operations keep pace with organizational changes.

#10 Security is easily overlooked

Once data is committed to a backup, that does not mean the data's safe.There is security to consider. Headlines are rife with stories of sensitive data slipping from the hands of couriers or being misplaced or even stolen. Since backups often contain confidential and protected information, companies must take pains to protect not only the principal data but the backups, too.

In fact, depending upon the industry within which the organization operates, legislation may require special steps be taken to protect backups from public release. When extending backup and restoration privileges and handling backup media, be sure that appropriate steps are taken to guard against unauthorized access. For online backups, this means ensuring the provider supports 128-bit encrypted data streams (and a separate encryption key for recovery).

19 comments
richca
richca

A few years ago, our company was next door to another that burned to the ground. In the evacuation, I walked out of our building confident that the numerous redundant sets of backups I had at a major offsite vendor would see our company through the impending disaster. Not only that, I knew our national vendor was capable of transporting those tapes to any of our other branch operations. Had I been lax in oversight of our backup operation I think I would have headed for the nearest plane out of the country - the company has annual revenues of $15 million going back several years. Because I had been following a lot of the recommendations in this article I was able to concentrate on implementing our disaster recovery program rather than planning an escape strategy. We actually were spared the destruction of what happened next door, but it is a cautionary experience I will never forget.

DPeek
DPeek

Backup (tape or other) as a system is no more vulnerable to failure than the very servers or apps that they are designed to back up. Either you plan, proceed, and succeed, or you go to the unemployement office with a crazy story about how the most unlikely thing happened to you...

InvisibleBoss
InvisibleBoss

Sure are many good points in the article. Still I really miss two tasks that should be critical to put into routines. First of all, there should be run periodically tests, that a RESTORE could be performed - and then check that the restored data actually is GOOD. A RESTORE could be done back to its initial place, OR to an alternative disk or folder. Regardless how much you put effort to the 10 mentioned points in the article, u need to know there is RECOVERABLE data on the tapes. The frequency, on how often restore tests are done, should be seen as how long back in time can u "afford" to have a miss (also seen as for how many backup sets are in use). As TAPES themselves (or other medias, if such are in use) also are worn during time. Tapes should NOT be trusted more than 18-24 months. Also, as a tape can be used several times, before it's full lenght/capasity is used, there should be a routine for RETENSION of the tapes (run/winded from one end to the other, and back) regulary. Even the BACKUP LOG should be read, to see if there are faults or errors reported for the job(s). And without getting completely paranoid, - at least Windows has its "Event viewer" - where "system" and "program" events can be scrolled for error messages. Here one may also get a warning abt possible "coming problems". Best of luck!! Tor Arne

InvisibleBoss
InvisibleBoss

Sure is many good points in the article. Still I really miss two tasks that should be critical to but into routines. First of all, there should be run periodically tests, that a RESTORE could be performed - and then check that the restored data actually is GOOD. A RESTORE could be done back to its initial place, OR an alternative disk or folder. Regardless how much u put effort to the 10 mentioned points in the article, u need to know there is RECOVERABLE data on the tapes. The frequency, on how often restore tests are done, should be seen as how long back in time can u "afford" to have a miss (also seen as for how many backup sets are in use). As TAPES themselves (or other medias, if that is in use) also are worn during time. Tapes should NOT be trusted more than 18-24 months. Also, as a tape can be used several times, before its full lenght/capasity is used, there should be a routine for RETENSION of the tapes (run/winded from one end to the other, and back) regulary. Even the BACKUP LOG should be read, to see if there are faults or errors reported for the job(s). And without getting completely paranoid, - at least Windows has its "Event viewer" - where "system" and "program" events can be scrolled for error messages. Here want may also get a waning abt possible "coming problems". Best of luck!! Tor Arne

Jason_Mcc
Jason_Mcc

This article is clearly full of anti-tape rhetoric, but what it doesn't provide is any recommended options?! If tape is so bad, what is the proposed alternative? Hard drive? And do people really rely on switching tapes by hand on a daily basis in a corporate environment? Any claimed 'reliability issues' about tape can be said also about hard drive. Of course I'm talking about current technologies here, not some relic from the 80's or something. If your backup is written good, it's just as likely to stay that way sitting on a shelf on tape as it is on HDD, for all practical purposes, numerical statistics aside. Hard drive has it's own problems, drawbacks, issues as well - can you put 600-700GB of data in your shirt pocket on hard drive? On an AIT4 tape you can, tapes are portable, light, durable and above all intended for repeated load/unload cycles. Hard drives are none of those things and are not intended for repeated cycling on a daily basis, not even hot pluggables. Above all, what other technology can manage itself? Combine a tape autoloader/robotic library with a decent backup management software product like Backup Exec or one of the others, and you hardly ever have to touch the tapes, perhaps monthly depending on your rotation schedule. I am not saying one is better than the other, for all purposes. Each has it's place - A proper, robust backup plan should include a combination of each technology - tape for off-site or infrequently accessed stuff, disk for stuff that doesn't have to move. Many people are trying to stuff backup-to-disk into roles it just isn't right for, simply because of the fact that the cost of disk has dropped far below the cost of tape per GB. But there is a price. Of course this is all IMHO.

DanLM
DanLM

What I don't think people understand is that data is shared ownership. System admins, application developers, users, and your customers all have a large stake in data being backed up properly. From my experience the sysadmin or operations were always in charge of performing backups, with all other individuals expecting this to have worked. I personally think that automated emails should be issued from the backup software to various delegates of each ownership group, other then the customers that is. This lets everyone know at least if a backup has failed, and it also insures that followup is done to either create a valid backup or to determine the reason for the failure. This is something that I would consider a no brainier, but I honestly don't see that happening where I am now. One person/department is responsible for running the backups while others just assume that it has been done. If priorities for the day stop this department/individual from checking the backup logs/email. Then nobody will ever know? Or, this person just might not consider it important enough to tell the appropriate individuals if the backup fails. Automated emails would remove this obstacle. Dan

jmurphyinsd
jmurphyinsd

It seems as though you're situation is accurate, but limited. Limited by the Symeritas software. There are enterprise storage providers that automate the entire life cycle that you just described with multiple copies of data (local LAN, remote WAN and multi-site failover availability). Symeritas just can't do that on disk. I would prefer to have a one step recovery (point n click) than a tape spin, load, extract, verify, cry, scream, shout and fail.

Endoscopy
Endoscopy

There are some basic differences in the mechanics of recording that make a difference in reliability. Tape makes contact with various parts of the transport and cassette. Disks are not touched by anything. As a result regular cleaning of the tape machine is a requirement as well as regular checking for head wear. A fact known only to tape specialists is as the head wears it works better until failure. This has to do with the gap and how it wears. Failure to clean the transport and replace cartridges often results in bad data. Failure to do maintenance of the transport as far as alignment and head replacement at the recommended hours of use can make a tape unreadable except by a purposely misaligned unit. I have done this to recover critical data. Disks do not have these inherent problems therefore are more reliable and less labor intensive.

apotheon
apotheon

Actually, disk-based backups are typically at least a little more reliable than tape-based backups, all else being equal. Tapes wear out faster, too. Finally, and perhaps most importantly, disk-based backups are a lot easier to verify than tape-based backups.

NOW LEFT TR
NOW LEFT TR

Disk to Disk & Disk to tape & Replication to DR Servers.

fdesnoyers
fdesnoyers

I use CDP solution from SonicWALL, no more tape or recovery failure. And it is a real-time solution, file modified is immediately sent to CDP appliance. In fact only modifications are sent to the unit, using less space on disk. Just click on version of file you want to recover and it is done. http://www.sonicwall.com/us/backup_and_recovery.html

NOW LEFT TR
NOW LEFT TR

Then every time something you are responsible for has an issue everybody who is effected should get an e-mail about it. If that's OK then you have no problems with the idea I guess.

ayazhoda
ayazhoda

I've worked with different places and what i've seen some companies investing huge amount of money with out investigating, documentation and error analysis. There are system error, human error, random error unless you have documented every thing correclty you cant able to reduce failure in backup. In addition my experience about tape drives are worst, its normally headache to change tape everyday pass it to security guy and put rest of the tape in proper save, now a days disks are good enough to trust on them.

thisisfutile
thisisfutile

That's a good point. I'm a tape advocate but because of the nature of my IT environment, I can't perform a verification because it takes too long. We're a one server environment (10 clients) and our nightly tape back-up takes 4 hours. An additional 3 hour verification just isn't possible because the server is already working very hard. Obviously a solution would be a different server or hardware configuration but that's just not in the budget. It could also be argued that verification is pointless in this environment because our back-up is what it is. If it failed it's not getting backed-up again anyway, so why verify a failure. Ultimately, the length of time it takes to do a tape backup and verification affects some IT environment's decisions.

sctang73
sctang73

When running a Windows environment, replicate your Windows server data, then backup from replication server into tape. Supplement w/ a high capacity NAS or iSCSI NAS for quick backup & restore jobs. Cost is unfortunately an issue for small environments.

jmurphyinsd
jmurphyinsd

Sonic, evault, iron mountain, etc., none of them can really support more than about 50gbs of data on any server. There are other providers out there that blow their doors off, and they're not EMC or Symeritas...

Sigman
Sigman

Such notifications are often necessary where databases are concerned. In my shop, my DBAs create local disk backups of their databases and the Network Admin group sweeps them off to tape/SAN/wherever. But the backup off-platform often fails without us getting notified, and nothing being done to rectify it beyond waiting until the next night's run. In one case, our off-platform SAP production server backup failed for two weeks straight. The Netguys didn't bother to tell us - I went looking and found it. Paranoia has served me well! I implemented scripted copy of the database backup files to another server to remove the single point of failure - and rode their butts until it was fixed.

thisisfutile
thisisfutile

I'll be showing this to my boss just to help emphasize the need for increased IT budget. Thanks for the, um, backup. ;-)

DPeek
DPeek

To Thisisfutile Youve got an appropriate nickname if thats the backup strategy you employ. No chance to verify? No window to work with? No additional hardware? I used to work for a storage vendor. People would spend all kinds of wild money of terabytes of disk, then have no strategy to back it up. Even before we started selling terabytes like hamburgers, when there was an emergency we would first ask about the existence of backups should the worst happen. Too often we got these responses: 1) "Do you know how LONG it would take to back all this up?" unsaid answer: Not as long at it will seem to take to update your resume and socre your next interview. 2) "Thats why we bought RAID! We dont NEED backup!" unsaid answer: RAID IS NOT BACKUP!! RAID is not likely to fail (depending on how you deploy) but the potential always exists. Plan accordingly.