General discussion


Catastrophic Success Story.

By AmberHaze ·
I just had an experience this week which I want to relate as it may cause some to stop and think about thier own situation.

We are all told to have our desaster plan in place, just in case the unspeakable should happen.

On Monday, I came into work to find that the front office was covered in six inches of water. (All the workstations are on standoffs, so none of them in this area experienced any water damage.) It turns out a toilet upstairs had flooded and had been running all weekend, pouring literaly thousands of galans of water into our sales office.

Unfortunately, the route it took to come down was via the wiring stack which is incidentally located right beside my server rack. You can just feel where this is going can't you.

Anyways, to make a long story short, one of my two servers decided to short out, blowing the motherboard, and power supply, and causing untold damage to other components.

Fortunately, my backup procedures include a 100% mirror of my LINUX servers HD's on a different machine in a different part of the complex. (If the server location and the backup location go out at the same time, I don't want any part of the re-build, as a good chunk of the town will be gone too...)

Anyways, I grabed a spare box of the shelf, thru in the backup mirror drive and booted the linux, had to fix one setting to do with the nic card and we were live as if nothing had happened.

Total down time... Opening + 45 minutes. In other words, fourty five minutes after discovering that we were down, we were back on line as if nothing had happened. (It took 2 days for the flood people to sort out the front office)

This is my first catastrophic event with a Linux server. I must say I have never had such a pleasant experience with comperable windows servers in the past.

I thought this would be a good launching point for a discussion on Catastrophic successes and fialures.

This conversation is currently closed to new comments.

Thread display: Collapse - | Expand +

All Comments

Collapse -

nice one

by Shellbot In reply to Catastrophic Success Stor ...

Congrats on having a good plan. Last place i worked in, IT was hopeless and we once ended up losing a small bit of stuff we held for another company..(we held several client that was embarrasing, thank god i was not with the team responsible for it at the time!

By the way, Prince Albert Saskatchewan..not a place i expected to be mentioned on TR :)

I'm a former resident, was just there for a few weeks last month visiting the family!

Collapse -

Wrong Config

by Grimshiire In reply to Catastrophic Success Stor ...

You could of had the same success if your Windows servers were setup correctly. Don't blame windows for poor planning. I have recovered a few times with windows servers that were properly setup. DR planning is the key.

Collapse -

You seem to have missed the point

by AmberHaze In reply to Wrong Config

Slow down there Grimshiire. The important point of this discussion is exactly the point of good Disaster Recovery planning. It was never intended as a comparison of technologies. Personally I manage many windows servers as well as Unix servers. They each have thier strengths and weaknesses.

In my personal experience, however, this was by far the best and simplest recovery I have ever done, largely due to the use of live remote mirroring which privided me with a drive image ready to use immediately. In the past, I have usually had to work with traditional tape backups which in a catastrophic situation require a fair amount of time and effort to restore the system to an operable status.

I think it would be very productive for people to relate thier good and bad catastrophy situations as we learn best from our mistakes and those of others.

Related Discussions

Related Forums