I just had an experience this week which I want to relate as it may cause some to stop and think about thier own situation.
We are all told to have our desaster plan in place, just in case the unspeakable should happen.
On Monday, I came into work to find that the front office was covered in six inches of water. (All the workstations are on standoffs, so none of them in this area experienced any water damage.) It turns out a toilet upstairs had flooded and had been running all weekend, pouring literaly thousands of galans of water into our sales office.
Unfortunately, the route it took to come down was via the wiring stack which is incidentally located right beside my server rack. You can just feel where this is going can’t you.
Anyways, to make a long story short, one of my two servers decided to short out, blowing the motherboard, and power supply, and causing untold damage to other components.
Fortunately, my backup procedures include a 100% mirror of my LINUX servers HD’s on a different machine in a different part of the complex. (If the server location and the backup location go out at the same time, I don’t want any part of the re-build, as a good chunk of the town will be gone too…)
Anyways, I grabed a spare box of the shelf, thru in the backup mirror drive and booted the linux, had to fix one setting to do with the nic card and we were live as if nothing had happened.
Total down time… Opening + 45 minutes. In other words, fourty five minutes after discovering that we were down, we were back on line as if nothing had happened. (It took 2 days for the flood people to sort out the front office)
This is my first catastrophic event with a Linux server. I must say I have never had such a pleasant experience with comperable windows servers in the past.
I thought this would be a good launching point for a discussion on Catastrophic successes and fialures.