After Hours optimize

10 Things Call for Feedback: What's your most memorable holiday-busting IT crisis?

For many IT pros, time off is hard to come by -- and it's often ruined by work emergencies. Share your worst (or most ridiculous) vacation experience.

Vacations and holidays... we all look forward to these times away from the hustle and bustle of the office so that we can recharge our batteries, clear our heads, reduce stress levels, and just be more productive. But during these times, you might be sitting on the beach or relaxing with your family when suddenly, your pristine surroundings and good mood are shattered by a most unwelcome sound: The work cell phone rings and doesn't stop ringing until you've answered it.

Of course, the caller wants to let you know that something has happened at work and your presence is desperately needed.

That's the scenario. Now you fill in the rest. Has this every happened to you around the holidays or when you were on vacation? What's the worst "Oh, @!%@" moment you've had while you thought you were going to be away from the office for a few days? You can share your story in the comment section. If you'd rather stay anonymous, email your story to tr10things@slowe.com. I'll select the 10 best stories and compile them for your reading pleasure (or horror!).

By the way, you can submit anything -- either your worst experience or funniest experience in this situation. I'll take both!

Additional reading

About

Since 1994, Scott Lowe has been providing technology solutions to a variety of organizations. After spending 10 years in multiple CIO roles, Scott is now an independent consultant, blogger, author, owner of The 1610 Group, and a Senior IT Executive w...

21 comments
a.portman
a.portman

At my previous employer, we were 24/7/365. It was a resort company. They are busy on holidays. I am on call for New Years day, which is also my wedding anniversary. I get a call. "We are out of toner. Can you run by your office and bring us one?" "Um, no." "Why?" "Your desk and the main IT office are in Orlando. I am in St. Louis." She finally believed me that going to OfficeMax, (I had to check that the Orlando one was open) buy a toner, toss the receipt in the drawer, print bills for everyone checking out and someone would work it out on the second, was the best alternative. My worst nightmare was actually the night before. Remote support of the cash registers in a bar in a Las Vegas resort on New Years Eve. Nothing happened, but I wasn't comfortable until dawn.

a.portman
a.portman

I work for a for profit college. We renovated the 4th floor of the building. This meant major electrical work. It was decided that Christmas break would be the ideal time, no students in the building. But, I need to be in the building while the servers are running on generator power. They run fine the whole time. Email flows, people at home can work remotely. Our other campus is up and running if largely empty. The generator runs the It room and its servers and lights. The boiler, elevator and non emergency lights, no. St. Louis, Mo in late December is in the low 30's. I only get "Dell" heat.

clarkcomputer
clarkcomputer

I was trick-or-treating with my daughter. We were both dressed in Army camouflage. I'm a veteran, she was wearing jungle and I was wearing desert. We both had camouflage paint on our faces. I got called to come into work at our California office because we had lost connection to our office in Panama. No choice but to go into work in desert uniform and painted face. It was a 7x24 call center so there were plenty of people there to see me. Fun times!

Slayer_
Slayer_

And so do our customers, so if it all breaks, no one cares till the new year.

sparky
sparky

Christmas day 2009 around 1:00 pm, I got a call saying the power to our building was out and I needed to go in to the office. The power company guarantees us 99.9% up time, and we have a generator, not to mention various UPSs so I had no clue how the power could be out. Well, on the way in to the office, I saw someone who had been partying way too much way too early in the day, had driven into a transformer which took out the power to entire block where my office is located. (I guess the power company didn't take DUIs into consideration). The generator failed to start, and of course the batteries didn't last the 6 hours it took the power company to replace the transformer. I was in the office until close to 10 that night.

technical.angel
technical.angel

After fighting tooth and nail, I was able to get the OK to upgrade our UC voicemail system, so it would be compatible with Exchange 2010. (Thank goodness the CIO is a Mac fanboi, and wanted to use the newest version of Office for Mac.) Since the entire university was closed from Christmas Eve until after New Years, we decided that the 27th would be the best day to do the upgrade that the tech was saying would take about 3 hours. 10 hours later, the Server Admin and I (Telco Admin) told the tech to go home, as he had about a 2 hour drive ahead of him. I ended up putting in about 35 hours during that week. 2 months later, during a conference call between the tech, our Server admin, and a level 3 or 4 tech at the man. of the voicemail system, and a tech at Microsoft, they found the tiny little setting on the Exchange server that was causing all the problems. Of course, there was nothing documented anywhere about this setting, and it's penchant to keep the voicemail system from communicating with it, until the entire voicemail system locks up and crashes. Great fun.

HAL 9000
HAL 9000

I hear that some others have them but everyone that I hear has taken one doesn't work in IT. I went to a Company Christmas Party and got hit by the Boss to fix his phone before I could eat though. Now if you push this buttom it turns it on like Magic. :D Col

Shrike49
Shrike49

Having just downloaded our medical certificates for our Travel Insurance, The almighty Crack of Thunder Exploded right over the top of my SOHO and as I rushed for the SHUTDOWN button a deep blackness decended upon the office AND I !! After switching everything off at the wall I headed for the Mains Power Board. All was well there, so I soon realised power was off to the Whole Area - Great!!. Some 30 - 40 Min later Power was restored to the area and the Lights came back on in my SOHO. BUT, My Computer would not Light Up at all! Checked this and Checked that ...NBG. Got onto the technical support outfit and comfirmed the PSU (Power Supply Unit) was NO LONGER a going concern!! How long before you can supply a PSU???...2 to 3 days was the reply! OK, can you install on sight? Not a problem, would be quicker if you can pickit up yourself!! Can't do that, I cant drive at moment for medical reason at the moment! OK will only be 2 - 3 days and a 200 Bucks. OK, beggers cant be chooser's!! Please Note, This is URGENT, I'm going overseas in a week and have medical files I need to get to my Travel Insurance Agent ASAP.!!! 10 Days later and probably as many Phone calls the PSU problem still had NOT been fixed and I Told the Outfit who was supposed to fix it they were F... Usless and Demanded a full refund!! Even then they took a week to do that! Meantime I was advised a Particular flight had been cancelled 24 hr before departure. After much mayhem, frustration and stress All was resolved and we Actually departed for our 1 month trip to China which turned out to be the ONE OF THE BEST Holidays ever!!! Should have stayed though!! Within 48 Hour's of my return my XP system had been repaired (by a small computer shop owned and run by a friend of a friend) and it was back to the grind stone!! Sort Of, but thats another costly story! Who's Need's Computers??? I survived (I think), without It!!! My SOHO has a seperate power Circuit and Surge guard protection for the Computer equipment.... Ha Ha HA!!!

larrymcg
larrymcg

At our company (Tandem, The NonStop Company) all the support folks and managers in development were on high alert and had to stay close to work in case something happened. As we all remember, nothing much happened.

kevin
kevin

About 7 years ago, someone took it upon themselves to decide that the business day calendar for our scheduling system really meant Monday - Friday (including holidays). So, about three minutes past midnight on New Year's Day, I got a frantic call from the data center telling me that dozens of jobs had just failed simultaneously. That's right, the entire batch cycle started running on a non-business day. Honestly, it didn't do any damage, but I spent about four hours making sure things wouldn't get any worse. And then I called the guy who changed the calendar at 4am to let him know what he did. He sounded like he'd had a fun night until then.

mhodgdon
mhodgdon

I took a vacation to Maine to do some camping and after being up there for only one day received a phone call from work that our T1 circuit had gone down (which handles the phones) so I made a quick phone call to our carrier (they knew us well as we had been having this problem regularly). The next day of my vacation I got another call from work that noone could get their documents located on the server and that they kept getting "IP address already in use" message from Windows. Luckily this was on a Friday and I did all I could remotely to get key people connected to their files. Come to find out that someone took it upon themself to re-arrange a room that contained a poorly placed network switch (that part is my fault) and had created a loopback in the system. Needless to say I quickly there after ordered new switches with loopback detection (we're non-profit so it's hard to get new equipment).

mike
mike

I run a from home business and I took my family on vacation after xmas last year. We were away 21 days and every day I would log into my server remotely to ck email, insure all my clients backed up, etc. Our last day, a Saturday, that morning I could not reach my server. I thought it was just my isp being out or a router issue. I figured since we were flying back that evening and it was a saturday, it could wait until our return. We arrived home at 10pm and while my wife fumbled for the house keys, the driver from the airport informed us that our front door was open. We proceded into the house to find that it had been ramsacked and all of our posessions stolen. I had just spent about $25,000 on new equipment for end of year appropriations and it was all gone. The perps had lived in our home while we were gone and not 1 neighbor noticed (or said or assisted police) anything. They even stole our vehicles as the keys were in the house. We had to live in a hotel for 2 months until our home was disinfected (bloodborne pathogens). They had totally wiped us out and broke whatever they couldn't steal. It took us about 6 months to recover. To add insult to injury, the defendants were juvies and only got 6mos probation for over $160,000 in damages and loss along with a piddly $1,000 restitution fine that if they don't pay, it will go away when they are 21. We are terrified to ever go on vacation again.

CharlieSpencer
CharlieSpencer

is that I've never had one! However, I can assure that Thanksgiving night can be one of the busiest night of the year if you work in a pizza joint. Man, do people get sick of turkey in a hurry! It leaves New Year's Eve or Super Bowl Sunday way behind.

omaha0428
omaha0428

Being in accounting software support, I've spent many New Year's Eve and New Year's Day holidays working to support our accounting users who just can't wait for business to resume on the 2nd - and seem to have more problems right then than any other holiday or vacation. Recently, some of my co-workers had a vacation issue though - all 3 support people were on vacation simultaneously (just asking for Murphy's Law to kick in) and they each were on the phone on a conference call until 2 am from their respective vacation sites helping to fix a critical problem. That is probably the worst one I've encountered (and I consider myself lucky!)

aaron
aaron

last year we had 2/3rds of the company (and my entire 5 man IT Team, of which I'm the manager). my boss and others picked on my for bringing my laptop and aircard. "We're on a dinner cruise around manhattan, lighten up." As it turned out, we made it through the evening without issue ... but had we not, how screwed would I have been?

OurITLady
OurITLady

At least part of the reason. The staff here aren't too bad at not talking shop at the party, but I still get any number of "my machine at home is doing something funny, can you take a look" - probably because they know I'll say NO in office hours.

HAL 9000
HAL 9000

[i]My SOHO has a separate power Circuit and Surge guard protection for the Computer equipment[/I] That's where you have gone wrong. That Surge Guard reacts how fast? There are Surge Suppressors and then there are Surge Filters that will stop a lighting strike at 100 Metres. The Surge Suppressers though are useless for that and by the time they trigger and dump the excess voltage to earth the damage has already been done so they start to react after the fact. The really good ones that stop a Lighting Strike from 100 Metres which is a big call are a 1 use only thing so after they have saved your Hardware they need replacing and as they are wired into the switch board they need to be replaced by an electrician which only makes it more expensive. You might even only have to pay about .5 of a K to buy one and have it installed properly. ;) As to the repair if you had of gone directly by the friend of a friend it would have taken how long? You also most likely got a better quality Power Supply as even a very high end Brand Name PS like an Antec comes in well under the $200.00 fitted. Col

robo_dev
robo_dev

It's a shame that there seems to be so much attention on catching perpetrators, but too often there is no real justice. Last year there was a home-invasion robbery in my subdivision. Two guy busted in the back door of a house on a Sunday afternoon with two kids in the house. The kids hid and called police, and several neighbors also called the cops when they heard the glass break. They caught the perps (all six of them). I actually helped since I had CCTV footage of their vehicles. Guess how much time in jail these guys got? Eleven months. They all had priors for drugs, burglary, and other crimes. Some of the crew even got probation. The root cause of some of this is that budget cuts, jail overcrowding, and judges who are often appointed to their job treat property crimes as a trivial offense. In this case there were no weapons involved, and nobody got hurt, although the police K9 unit dog got a nice set of teeth on one of the perps backside. :)

seanferd
seanferd

Way beyond an IT issue. I'm sorry, that is completely abominable. So, even with no witnesses (somehow), they caught the perps and still nothing happens? Ridiculous. An ongoing crime like that should have relieved the defendants of any defense of youthful indiscretion.

HAL 9000
HAL 9000

They get treated much differently to adults and the younger that they are the less likely they are to get a nasty punishment. Depending on which country you are in you can even get away with Murder if you are young enough. Here it's anything under 10 years of age the perp is not responsible for their actions. It can also give you a great laugh where some 8 year old here was removed by the Social Diseases from their parents because the parents where way to strict on the child and where unreasonable. Of course the child had not been in trouble but had complained indirectly to the Social Diseases of not having any ability to do as they please. 3 months latter the child stuck a knife into the Social Disease when they told it that they couldn't do as they please and where promptly returned to its parents who could control it. I really thought that was Poetic Justice but it unfortunately doesn't happen anywhere near often enough particularly as in todays beliefs [b]Children Never Lie[/b] and are always abused. The possibility of any child being a [b]"Proper Little Bastard"[/b] just isn't possible, but there was another one who went looked at a second hand bike said that they liked it and then proceeded to beat the seller to death with a fence post and stole the bike. Both cases the kid was under age and never charged with any offense and interestingly both where female. The boys on the other hand just steal things and do less damage from Police Records here at least. ;) However I don't know if it's still going on but about 15 years ago here the Major Drug Dealers where employing children under 15 years of age to do their Drug Deals for them. One kid got caught with half a kilo of Heroin and not only was let go but had the heroin returned to them by the police. Of course it was hard to realize that anything was happening as the kid in question was always in a Limo being driven around and generally showing off spending bulk amounts of money. I back them called it the Fabian Experience after Oliver Twist. It made perfect sense to the Bad Guys who couldn't get Prosecuted as they where never caught with anything illegal. Col

jpnagle59
jpnagle59

...that just ain't right, not right at all...it seems as if parents are now 'left out of the equation' on raising their kids...my son pulled shenanigans when younger, but he knew what line not to cross, because he then would have to deal with the man-and that be me! Parents do your kids right- you have to give them 110%, or more to get them down a good road to life...