Desktop

General discussion

Locked

Good Disaster Recovery War Stories: Got Any?

By robo_dev ·
I've worked through four pretty significant IT disasters over the years.

The most interesting and avoidable one was the following:

There was a simple move planned for a computer mainframe system from one data center to another on the same campus. Easy, right? What could go wrong?

The computer movers showed up at 6AM on Saturday, and quickly loaded the 18-foot truck with the processor units, controllers, tape and storage systems, no problem.

One tiny detail was missed.

One of the movers stopped to have a smoke.

He was the one who normally closed and latched the back door of the truck.

As the truck pulled up the loading ramp from the loading dock, the rear door swung open, and several million dollars worth of IBM disk arrays slid off the back of the truck, falling about four feet to the pavement.

A big 'oh shoot' moment.

24 hours and a restore-from-tape later, things were working.

This conversation is currently closed to new comments.

Thread display: Collapse - | Expand +

All Comments

Collapse -

I have mixed feelings on that

by NickNielsen In reply to I wish our customers woul ...

As the on-site tech, I'm in favor of anything you can do to reduce the amount of equipment I support. On the other hand, the ASP model inserts another layer, making outage response exponentially more complicated.

And it doesn't change the fact that I still get called when the system goes down. Why?
- The ASP said "Our equipment is good, and the servers are up and running. It must be the network."
- The ISP said, "Our link checks good from demarc to demarc. It must be an on-site problem."

Collapse -

We only provide ASP...

by JamesRL In reply to I have mixed feelings on ...

.....when we own/manage the networking equipment on site, and when the client contracts with an approved ISP with a QOS guarantee and what we consider "business class" service.

If you are too cheap for that, you have to buy your server from us and locate it on your premises.

We have some pretty heavy duty network engineers and some decent tools. We rarely have to fight long with an ISP.

Collapse -

RAID in Flight

by GSG In reply to Been there....

I had to buy a seat on a flight for a RAID controller once. I picked it up at the airport and was met by the flight attendant, pilot, co-pilot, and half the people on the flight all of them wanting to know what was so precious about the box.

There was just no other way to get it here as quickly as I needed it.

The hardware had been purchased straight from the software vendor, and they had to foot the cost of the flight. However, we did tell them that from then on they had to provide us extra on-site spare parts for that server.

Collapse -

Yup, just had to do that for some SAN hard drives

by robo_dev In reply to RAID in Flight

I wonder if they offered the box of Hitachi drives a coke and some peanuts as they strapped them into the airplane seat?

Collapse -

Message has been deleted.

by aoomall In reply to Good Disaster Recovery Wa ...
Collapse -

Waterfall

by GSG In reply to Good Disaster Recovery Wa ...

Our biggest disaster was the waterfall that started on the top floor, and ended directly in our server room.

A private office on the top floor had its own bathroom. On a friday afternoon, the toilet started leaking, so instead of calling maintenance, they put a bucket under the leak, thinking that since it was a drip, no problem. Yes, problem. The pipe broke and filled the office with water, then it started filling up our server room.

We found out when the servers started going down and we started getting pages. We lost about 10 of our servers, but were able to salvage enough parts out of the servers, use spare parts, and servers that had been removed from production to cobble together the most critical servers. Luckily, our insurance approved us to get servers right away and worry about cost later.

We called one of our vendors and they special shipped servers that got here less than 12 hours later. Less than 24 hours later we were done.

That episode was a great test of our DR plan, and instead of admin and our users being angry at any delays, they commended us on getting everything back up so quickly without any data loss.

As a side note, that bathroom has been gutted, and is no longer a bathroom. A year after the flood, a new department head put a request in to have that space made a bathroom. His request was quickly denied, and the blue prints marked with "never put a bathroom in this space".

Collapse -

No floor drain?

by NickNielsen In reply to Waterfall

:0

I can't imagine putting any kind of plumbing facilities in a commercial building and not installing a floor drain, executive washroom or no.

Related Discussions

Related Forums