Discussion Locked

Oops! IT technician wipes out $38B fund

0 helpful
13 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

http://www.usatoday.com/news/nation/2007-03-20-alaska-fund_N.htm?csp=34

The first words out of my mouth would be M.F. Then, S.O.A.B. I think that would continue for like 30 minutes until I got up the guts to go to the supervisor. When the backups are found to be unreadable, I would just resign I think and get out of the Tech business.

God, can you imagine how bad that person felt?

Lesson learned, test all backups. Wonder why the tapes were unreadable? Bet they were running them for years without ever running a Disaster Recovery test.

Dan

13 replies

by stress junkie

In all the years that I worked in large businesses I was never able to convince anyone to perform fire drills for our disaster recovery plan. In fact as far as I know most places that I worked didn't have a disaster recovery plan. I know that none of the IT people were ever given any information about disaster recovery. Managers and coworkers always had better things to do on the week ends than to see if we could recover from backups. At least I tried to keep our tapes in good order by reading backup error logs and trying to read the backup data from the tapes. I threw many tape cartridges away that had been in service for many years.

You know I have often heard people say that a good manager will hire talented people and trust them to do their job correctly. It seems to me that a good manager will check to see if jobs are being done correctly and will take the initiative to fix whatever isn't being done well. Most of the managers to whom I reported over the years didn't even know what I was supposed to do, much less if I was doing it well.

by tony hopkinson

Run back up, throw tape in cabinet, then jump off the top floor, when they find it didn't work.
Learnt the lesson the hard way these fellows.

I started in a new job and was just learning VMS. So I read up and discovered that the last three years back up tapes were useless almost by accident. If I had just followed the written procedure in place, it would have been a deep poo situation, a few months later the hard drive failed.

Well worth getting a third party in for a sanity check on these things.

Rule One, you haven't successfully backed up if you haven't restored it.

Rule Two, it may not work next time

Rule Three, you'll never need it unless you really really need it, so it's not worth doing right.
Huh?

by locolobo

I learned that the hard way. Fortunately it was only a small backup (3000-4000 old files). But when we needed to review one of them it was unaccessible. Since then we doa small recovery test for every backup on that particular system.

by tonythetiger

[i]Bet they were running them for years without ever running a Disaster Recovery test.[/i]

As data grew and backup windows shrank, one of the first things to go was the verify option on the backup.

I'll bet the guy's still in hiding :)

by dspeacock

that while most businesses perform backups, and only a portion of them test those backups, how many businesses (in these days of SOX, GLBA etc. where retention is required for 7 years) keep a record of the actual system configuration the backups were made on, just in case they have to restore a really old file? Even better, how many have a record of the administrator or user passwords for those systems?

It's all well and good to have viable, tested backups, but if you haven't got the system or a record of what the system was, a backed up file is about as useful as screen doors on a submarine.

by tony hopkinson

I remember restoring sql server 6.0 databases after we'd re installed the OS.
We were using integrated security and the SID mappings went completely out of whack. repluser was sa, sa was me ...

Things like NTFS file permissions, encryption etc can be a right performance to find. Things like DCOM config could generate one or two belly laughs a month or so after as well.

by mjd420nova

Decent backup software should verify what's written as part of the finish procedure. I always use two sets of disks, one a week old and one two weeks old. That way when you do a weekly backup, you overwrite the oldest (two weeks) set and verify after writting that it is good. That way if the vefify fails, you still have the most recent backups still intact. That's when I find most failures of hardware is, during the verify after writing. This applies to disks, tapes or even in network backups, never overwrite your most recent copies. A recovery test should be done monthly or anytime there is a hardware failure or upgrade to the backup hardware. That saves a lot of work and maybe some jobs. Even in the event of complete destruction of data and hardware, data could be reassembled from their original sources and new copies made from that. Only a major failure would wipe out all, such as a major fire or tornado, which I have been unfortunate to experience both. A real nightmare but reconstruction is possible, at least the data anyway.

by dspeacock

is nice, and generally works, BUT like all things electronic or mechanical, it will fail. Turn it on, yes, but DO NOT skip the actual testing of random backups at random intervals. If you need your backup data in a hurry, you won't be sorry.

by tony hopkinson

Doesn't mean you've backed up what you needed to back up, only that what you backed up was read and compared once.

by danlm

Lol, I was just thinking about that. Any time someone here's me go oops... roflmao, they usually check to see what is wrong.

Dan

by paulaaa14

I'd like to know how far they looked into recovering the data from the formatted disks. Normally it can be done. OnTrack? would it have cost more than the $200K they spent? Just curious.

by why me worry?

Microsoft and Dell don't know jack sh-t about data recovery, so why these morons paid for MS consultants to come in and waste their time and money is beyond anything reasonable. They should have immediately went to Ontrack Data Recovery if they wanted to bring their data back. Hell, Microsoft can't even crack their own EFS encryption for those who stupidly lock themselves out of their encrypted files, and one expects them to be able to recover data from a formatted hard drive? Good luck cause it aint gonna happen!

by why me worry?

and no DR plan will be put into place because:

A. They lack the skills and expertise to architect and implement an effective DR plan

B. They lack the funds to implement high end SAN storage arrays that replicate to offsite locations

C. They assume this was a one time incident and can never happen again

D. Nobody wants to give up a weekend to perform DR testing to ensure that tapes and hot standby systems work in the event of a disaster

I speak from experience having worked for both public and private sector organizations

This discussion is closed to new replies.