Disaster Recovery

Reviewing my backup strategy and a backup horror story

Every now and then, you should review and revamp your backup strategy. Joe Rosberg explains his backup and storage process for an architectural firm that includes hard disk drives rather than tape.

Every now and then, you should review and revamp your backup strategy. Joe Rosberg explains his backup and storage process for an architectural firm that includes hard disk drives rather than tape. 

-------------------------------------------------------------------------------------

Every so often, I review and revamp my backup strategy. Although many would disagree with me, I've avoided tape backups for years. Instead, the media of choice for my backup files is the hard disk drive. It's simple; the files are easy to find and retrieve; and it can accommodate a multi-tiered backup strategy. (However, I will acknowledge, that providing a backup strategy for 25-30 people is different than doing it for hundreds or thousands of users.)

While I can't back up every iteration of every file, I do provide a daily backup, a weekly backup, a monthly backup, and a longer term quarterly archive saved to DVD; in addition, I provide an off-site backup by way of an eSATA external hard drive enclosure that I rotate weekly. (Maybe I'll start doing this daily, and perhaps even do it over the Internet.) Hard drives are very inexpensive, and the capacity is getting larger all the time. I can easily put every file in the office onto a 500GB drive and still have some room to grow. As that need does grow, I'll just start using some 1TB drives (currently available for a very low price in the $200 range).

Being in the building design industry, my users often want to go back and retrieve an older version of a design file, wanting the version that might be a week or more old, even though the file was changed and updated daily. I'm probably about 95 percent successful in being able to retrieve the iteration they want, realizing that some will simply fall through the cracks. I simply copy the files from Point-A to Point-B by way of running a number of scheduled batch files. Spread out over a half-dozen hard drives, I can save several versions of the same file, but saved from a different point in time. Although it's not perfect, the system has worked pretty well for me.

Oh yeah, the horror story. Luckily, it's not mine.

I recently ran across a news item that many of you have probably already seen: one about an employee at a Florida architectural firm who saw an ad for a new hire, listing the phone number of her boss. Thinking she was about to be fired and replaced, she went into their office one night, logged into their network from her own computer (no one claimed she was smart!), and deleted all the files on the network. It turns out that her boss's wife placed the ad for a different company, and that's why the phone number was the same. Suffice it to say, she did indeed lose her job, although it was not in jeopardy before this incident.

Her boss has said that she deleted seven years worth of drawing files, valued at upwards of 2.5 million dollars. He went on to say that he was able to retrieve those files, but only by paying for an expensive data-recovery service. Wait a minute, I thought. If that happened to me - and I am in the same industry - I would have had those files on a backup drive, one to which no one else has easy access, not to mention my off-site backup which is never more than a week old. (This is what's made me consider taking my off-site backup out on a daily basis instead of weekly.)

I am amazed that this architectural firm didn't have a better backup strategy.

Do you have any backup tips that you'd care to share? Any horror stories about backups?

36 comments
JohnMcGrew
JohnMcGrew

Selling clients on the vital need for a tiered and rigid backup regimen is hard enough. But then having to sell them drives that could cost $500, $1000, or more and on media that could cost $50, $100, or more each was just too much. Then, you???d have to convince them that the media was only reliable for so many years, and should be replaced long before errors started to appear. And of course, you???d only find those errors if you did regular checks of the tapes, which also cost the client money. And then again, being relatively sensitive mechanical devices, I rarely had a drive that would work reliably for more than 3 or 4 years of regular use. In over 20 years of use, I never felt comfortable with the reliability of tape, and it just didn???t make sense that I???d be spending more on tape drives and media than I???d be on hard storage in the first place. And the gap between cost/gigabyte for tape vs hard drives kept widening. And all of this expense was on something that everyone prayed that they???d never have to rely on. A dozen years ago, I flirted with optical for awhile, but was never satisfied with the speed and the need to media-swap in the middle of backups; something you couldn???t trust clients without staff dedicated to the purpose to do. And the reliability of optical was only marginally better than tape, especially for RW media. About 6 or 7 years ago, I finally converted most clients away from tape to removable hard drives and never looked back. The reliability (both long and short term) is no worse than tape or optical. Data and hardware failures are usually discovered at write time vs restore time, and the cost/gigabyte is practically the same. Backup times are faster. They should last at least as long as the drives you are backing up. And since they are so cheap, you can afford as many multiple sets to rotate backups as much as you desire. They are no more trouble to move and store off-site than tapes are, and hold more data for the same amount of physical space. Other than for legacy purposes, I can???t imagine why anyone would want to spend $500-$1000 + cost of media for a tape drive that can only slowly (and serially) write dozens or hundreds of GB when TB hard drives can now be had for

admin
admin

Agree with the hard drive strategy. Ideal software for this is SecondCopy (.com) Currently have: backup to separate partition on main hard drive backup to separate hard drive (internal) backup to external USB hard drive #1 (stays in the office) backup to external USB hard drive #2 (leaves the office whenever I do) backup to 8Gb USB memory stick (always in my pocket) Also considering core files backed up on secure web space. You can't have too many backups! Theft/fire/HW failure/sabotage/satellite crash! All of this is done automatically on a 2 hourly schedule using SecondCopy and changed copies are archived indefinately. Also - the backed up files are still accessible as actual files - not compressed data sets.

admin
admin

Agree with the hard drive strategy. Ideal software for this is SecondCopy (.com) Currently have: backup to separate partition on main hard drive backup to separate hard drive (internal) backup to external USB hard drive #1 (stays in the office) backup to external USB hard drive #2 (leaves the office whenever I do) backup to 8Gb USB memory stick (always in my pocket) Also considering core files backed up on secure web space. You can't have too many backups! Theft/fire/HW failure/sabotage/satellite crash! All of this is done automatically on a 2 hourly schedule using SecondCopy and changed copies are archived indefinately. Also - the backed up files are still accessible as actual files - not compressed data sets.

goorsha
goorsha

On the subject of file backup, sharing and storage ... Online backup is becoming common these days. It is estimated that 70-75% of all PC's will be connected to online backup services with in the next decade. Thousands of online backup companies exist, from one guy operating in his apartment to fortune 500 companies. Choosing the best online backup company will be very confusing and difficult. One website I find very helpful in making a decision to pick an online backup company is: http://www.BackupReview.info This site lists more than 400 online backup companies in its directory and ranks the top 25 on a monthly basis.

G-Daddy
G-Daddy

I think you've created a solution that's tailored to your environment. I have a few basic things that you may want to consider: a) How old are the disk drives you're using to store data and do you have a service retirement plan in place? b)Do you perform any maintenance on those drives? c)What happens in the event a specific drive has a physical failure and the data is unrecoverable?

Joe_R
Joe_R

Re: the original piece. Do you have any backup tips that you'd care to share? Any horror stories about backups?

shamit
shamit

It was intersting to see a writeup on an architectural firm's backup policy. I run my own little firm and take backup on my own. I have always wondered the safety of having HDD backups for two reasons. 1. Everyone has had or will have a HDD crash. 2. More crucial - What if a virus has managed to sneak past into the backup and keeps infecting every time you restore. Arn't CD's / DVD'd safer.

doug
doug

We currently use tapes because I can take the latest backup home every night, and if there's a disaster we can be up and running in a few hours on the new equipment. We're exploring USB hard drives, but how do you protect them while traveling? And do you only buy certain kinds? We're using Rsync because one of our servers is Linux.

Joe_R
Joe_R

All options are worth considering

doug
doug

Well, this is where something like truecrypt comes in handy. Instead of having your company financial information sitting in an excel file on another server, all that would be there is the encrypted truecrypt container.

Joe_R
Joe_R

When you're talking about a significant amount of data (into the many GBs), connection speed is the biggest consideration. Not to mention the security aspect.

Tharg
Tharg

Being in the consulting game I advise clients to use at least two backup vectors. On site Disk to Disk (encrypted NAS boxes are perfect for this) is preferable to tape. I also recommend online backups - am not US based and speed/cost is an issue here. Last option is tape backup - if client still wants this after I've given them many the stats on tape restores!

JohnMcGrew
JohnMcGrew

IMHO, the odds of a drive-data failure for removable hard drives is certainly far less than that for tape. And they certainly should last at least as long. So my answer to your questions would be: a) Certainly the same program than I'd be using for tapes b) No more than I'd be using for tapes c) The same than I'd be using with tapes

Joe_R
Joe_R

That's the key. What might work for one, might not work for another. But that's the great thing about these kinds of discussions. They open the door to considering other peoples' ideas. Thanks.....

StealthWiFi
StealthWiFi

I had a similar setup in place for a small company. They were running 2 servers and aproxamitly 7 high work load users. Every Month both servers had full backups which were moved to Blu Ray Disc and imideatly offsite then onto a hardrive secured in the office for fast recovery. Daily incremental backups were made and moved to the drive for storage then moved to a Blu Ray or DVD disc at the end of the month. The email portion of the exchange server was fully backed up weekly and one quarters worth put onto a Blu Ray Disc. There database files were backed up daily and moved to the drive and disc. Also Shadow copy was run on the File Server 2x a day and all useres were tought how to recover there own files using it. The Shadow Copy was kept for 1-2 weeks depending on request.

RNR1995
RNR1995

I have been using hard drives for backup since 1995. Once you use them you will never go back to tape. Tape is slow and unreliable. The easiest solution for business is a second set of hard drives on site, with removable drives. If you are using Windows, shadow copies work so well I swear M$ must of stole it from somebody, their stuff never works this good. You can use an old server as your second storage area, or use a NAS. I like the two servers, as you then can have a backup everything, Domain Controller, DNS, WINS, DHCP etc. Even for a small business it is really inexpensive...What's your data worth? That firm in Florida should be ashamed of itself, how could one user delete everything?

rkuhn040172
rkuhn040172

We use regular old USB drives, 320 GB versions for off-site, 750 GB for on-site. Cost $60 right now with eSata and USB ports for the 320 GB, more for the 750's. Daily backups on-site, incremental. Weekly backups off-site, full. Monthly backups off-site, full. In addition, we use shadow copies for user convenience. I will never go back to tape. HD's are faster, cheaper and easier to use.

SKDTech
SKDTech

that horror story hitting the news as I live in Jax, FL. My first thoughts when I heard the newscast were to the effect of A)They didn't have a single backup? B)She is even more screwed now than if they had been seeking a replacement for her. C) Why did she even have such broad access to the network and D) Who were they employing to take care of their IT infrastructure?

groenem
groenem

Acronis TrueImage creates a hidden partition on your HDD, making it impossible for viruses to access it. Also, if your OS crash and you cannot boot up into Windows, it has a Emergency route, that bypasses the Windows OS and uses its own OS from which you can restore from. Just install it, set up your schedules and forget about it. Just make sure you make a backup to DVD every now and then to cater for HDD crashes.

JohnMcGrew
JohnMcGrew

For one thing, if a drive is about to fail, you usually get clues long before it actualy does. When a tape goes bad, you usually don't discover it until the worst possible moment; when you need a restore critical data from it. Also, so what if a drive does go bad? They are so cheap that you should be running multiple sets. For most clients, I use two, sometimes 3. That means at least 3 shots at getting data back. And that is in addition to the other forms of backup I use, which include on-line and alternate-server backups. Considering all of the inexpensive options now available, there's little excuse not to have multiple backups of all of your data available, and at multiple locations.

Joe_R
Joe_R

Won't viruses still get copied?

admin
admin

Yes there probably are advantages to having backups on DVDs. My policy is to have as many different TYPES of backup as possible / practical. DVDs - USB Memory Sticks - Internal hard drives - external hard drives - on-line backup. They all have their weak points. Its the combination and some duplication that provides the security.

The 'G-Man.'
The 'G-Man.'

Evey Time! D2D2T From RAID Array ... Copy this data to SATA disk ... Copy data from SATA to TAPE RAID Array Fails Hot Spare, Rebuild or Copy back from SATA or Copy Back from Tape (worst case) Also - System Images - System to Disk to Tape

Joe_R
Joe_R

Either way, it can still be taken home every night. The tape has to be restored. The hard drive, on the other hand, could actually be put into a server, and you're good to go.

admin
admin

Personally I use a Maxtor 160Gb OneTouch III to take out of the office. Its based on a 2.5 inch laptop drive so it is small enough to go into a pocket or bag. If I was really being cautious I guess I could put it in a sponge lined box - but portable drives normally have some degree of shock resistance built it. So if I get hit by a truck the drive would be broken perhaps - but not the USB memory stick in my other pocket!

Joe_R
Joe_R

Isn't a Blu Ray writer pretty pricey? What about the discs? Is there a cost-time benefit over DVD?

gavinhughes86
gavinhughes86

"If you are using Windows, shadow copies work so well I swear M$ must of stole it from somebody, their stuff never works this good" rofl

Joe_R
Joe_R

I also gave up on tapes in about '95. At the time, people called me nuts. But I was setting the stage for a much better system.

Joe_R
Joe_R

.....is the shadow copy (I assume you mean of the user's computer). I've thought about it, but I like it better when I can say to save all important files to the network, because user computers are not backed up.

Joe_R
Joe_R

I looked for a more detailed story, but couldn't find one.

Lovs2look
Lovs2look

Disk to disk backups daily (o/night) and several generations worth, then once a week to tape. With the low cost and speed of HDDs this makes sense these days. Also we never backup individual PCs or laptops as any data worth saving goes onto a network drive/location. This is stated in out policy too so if a user loses personal files (audio/video/emails/whatever) then our IT dept. isn't wasting time finding and restoring non-business data. We have caught out a few users with this one...Hard disks WILL fail...it's inevitable. Pretty soon they learn to live without it, or they don't store personal files on work computers anymore...YAY!

doug
doug

I'm still unsure if you can take them home every night. Hard drives aren't very tough.

StealthWiFi
StealthWiFi

A good deal on tape systems can be found but a nice one can run around $1k plus tapes. If you look around you can find a great deal. This is the one we use and it works great and only $280.00 http://www.bestbuy.com/site/olspage.jsp?skuId=8844673&productCategoryId=abcat0504008&type=product&tab=2&id=1208562592208#productdetail Disks are pricey I am hoping they go down in price but 50GB RW is around 50 - 60. With a good compression program (WinRar is great) you can compress around 40% of the size without much risk. I am personally not a fan of tape backups becuase they are very isntaable and easy to break. A DVD/Blu Ray will not decompose and Blu Ray is very scrath resistant and durable.

rkuhn040172
rkuhn040172

Servers only. Only because I've trained users how to retrieve their own files instead of taking my time to retrieve them from backup. Otherwise, yes everyone saves to network drives.

Editor's Picks