Discussion on:

10 holiday-busting IT crises

The one about the cold Christmas reminded me of one at a university in Louisiana back in the 80s. The state was suffering from severe money issues and was forcing all state agencies, including our university to cut budgets. To save money since no one was there for the holidays they shut off all of the HVAC in the administrative tower where we had the entire basement for our data center.

Then Mother Nature took over and it got so cold the water in the HVAC pipes in the library on the second floor froze and burst the pipes. After the thaw of course water started flowing down to our floor. Fortunately the break was not over the computer room but there was standing water from the hall running into the raised floor through the door, the operations supervisor used boxes of punch cards to build a dam to stop the water from going much past the door. My office was the low corner so I had about an inch of water standing in it and anything on the floor was ruined.

A few years ago my boss had a spare ticket for a New Years Cruise. While I normally prefer the week of him away as an extra vacation, why not. The day we were to go to Haiti the weather was miserable, so we just cruised slowly past. His Cell (yes he had it on) goes off and the print server is down. I turn my phone on and head to the computer center. 1 hour later they are back up, yes you do have to yell from that far away.
Stress on a cruise, oh and one hour of cruise ship cell and computer time, that was about $600.

... with the ironic exception of Y2K. Good thing I celebrate on the 21st or so.

What moron thought that using the holidays at year's end is a good time to do major upgrades? Yes, I understand the whole write off business expenses thing before year's end, but tied to that is everyone else's crunch time with reduced manning. If anything goes wrong, IT and key department people are going to be working some long hours with disappointed children.

A little foresight earlier in the year could solve this issue. To counteract it, for the months of November and December our organization has an upgrade freeze. Hardware, infrastructure, and mission critical software do not get touched other than backups and regular maintenance until January.

Does it work? Sorta. We quit working the holidays internally, but it would seem that many of our clients have not received this memo.

Wasn't a DUI that took out power but a hot Australian summer day caused the local transformer to overheat and catch fire.

We also have a backup generator and UPS in the server room, however one of our outdoor staff had decided after the last generator test to unplug the generator battery recharge unit and plug in a cordless drill charger. The generator didn't start and the UPS ran down.

Needless to say the fix was a steel cage around the generator battery charger!

Working in IT for more than 15 years, help me to understand the nature about the vacations, christmas and any other days off for IT Manager and admins.

Every single time I go to vacation, I need to get my laptop with 3G internet connection with me, just in case. I had 3 or 4 ready-spares computers at the office so if something happens with a user PC, I instruct the guy at the fileroom just to replace the computer.

I have almost everything redundant in the office (just in case) - riverbed accelerators, vmware, dhcp, dns, ad, BES. Again, just in case.

I had vacations where I was in another state and the air conditioning system in the server room fails. I need to go back to the office just to reset the HVAC system (sunday night) so now, when I go to vacations, I have spare air systems.

The same with the power in our building, we have a Big Symmetra UPS and I configured notifications so when power go out, I get notified. If power is no back in some time, I'm able to connect remotely and shutdown non critical equipment. This allow me to have batt time for servers and network equipment for about 6 hours. If power do not return for some causes, I will need to shut down everything and users will need to use Message One from Dell. This never happens but you need to have a plan for everything.

We found out later that the generator's fuel filter was gummed up for some reason, so that's why it wouldn't start. And since then, yes. It has been religiously serviced and tested.

I wish I had seen the original call for stories. It seems that sparky and I are comrades-in-surprise. Exactly one year earlier, I had nearly the exact same story. A DUI took out a transformer on Christmas morning. Our UPS maintenance had lapsed and one bad cell vented. When power was restored, several blades balked after the un-conditioned jolt made it past the failed UPS. Thankfully I wasn't the only support working on the holiday. Perhaps we will meet one day and share a beer over the coincidental stories.

Thank goodness I wasn't considered essential personnel. One evening a sewage pipe on the first floor broke. Nobody was around all weekend so the flow just kept going. When we walked in Monday morning, the smell was horrible. We found where the sewage broke through the suspended ceiling and traced the pipe back to the break. The clean up was nasty but we finally got management to understand why it wasn't a good idea to route that sewage line over the computer room.

I scheduled a vacation day for moving from a rental to our first house (it came with a mortgage :P). Unfortunately, our rental was within easy walking distance from work and the morning of the move, my supervisor came knocking, saying the Internet was down. Turned out my senior (age-wise) colleague who was supposed to act as my backup didn't know the difference between power-cycling and hard resetting a router/gateway/firewall. Sigh... Thankfully, I had a recent backup of the router configuration, so it was a relatively minor distraction.
By the way, this is not a case where the backup "technician" became one by default/accident. This person has a graduate-level computer-related degree.