Have you ever had one of those things happen to which you said, "Oh no, not that and especially not now"? Do the worst things always happen at the worst times, or does it just seem that way?
When those things happen, it almost always starts out as a normal day, and this one was no exception. It was actually a nice change of pace for me, since I put my IT hat on the shelf and was involved in getting a design project out the door to meet a deadline. We finished the design drawings in plenty of time to meet our 3:00 deadline, and I was looking forward to going out for a birthday dinner with my wife and son. I had to go home to put some finishing touches on my Wednesday blog piece before I posted it, after which we'd head out to celebrate my big 5-5.
At about 4:50 one of my users came up and said he lost his e-mail and Internet. I quickly checked my own Internet connection and found that I lost mine as well. That's not really unusual, and I figured I simply needed to recycle the modem to reestablish the Internet connection. As I was doing that, someone else came up and told me that he lost his connection to our networked drives. After checking a couple of other computers, I found that everyone lost their mapped drives. The servers were still up and running, and everything seemed to be okay, but I went through the usual reboot process anyway, hoping that would reestablish the connections. No such luck. Oh great, I'm getting ready to walk out the door and the entire network just went down.
On one hand, since it was the end of the work day and everyone was leaving the office, it was probably the best time for a network failure (and I was certainly glad it didn't happen as we were pressing to meet that project deadline). On the other hand, however, I didn't want to stay after hours to figure out why it failed -- it was my birthday, for Pete's sake, and I was really looking forward to that steak dinner (not to mention the time I had to spend to meet my Wednesday blog deadline). And being a one-person IT department, I couldn't ask anyone else to do it. Oh well, let's just get 'er done.
At this point, not knowing the cause of the failure, I thought the best thing to do was to literally shut everything down and bring it all back up one piece at a time. So I walked through the office, shutting down everyone's computer and the networked printers, shutting down the servers, and powering down the router, modem, and switches. There were a few people still around fretting about losing some of their work, so I helped them save their stuff to their local drives and sent them on their way. At this point it was already well past 6:00, and I was resigned to the fact that my birthday dinner wouldn't be that mouth-watering, medium-rare steak, but rather take-out Chinese food. (For some reason, take-out Chinese seems to be the food of choice for after-hours IT work.)
After bringing up the router, modem, and switches, I turned on my main server. The first thing I checked was for an Internet connection from the server -- nope, no go. I went through that same process a couple more times, but all with the same result. This did narrow down the problem to one of three things: a bad modem, a bad router, or a bad switch. I then bypassed the switches and connected the router directly to the server, which did indeed give me the Internet connection. Going back through a switch, I lost it. I have two switches, and regardless of which one I went through, it wouldn't work. What's the likelihood of both switches failing at the same time, I wondered?
It was now close to 8:00, and the only place I could get a couple of switches was at the Micro Center store way across town, which closed at 9:00. I suppose I was fortunate that I still had time to get there. With two new switches in hand, I decided to isolate my old ones before replacing them both, and it turned out that only one was bad, but when they were linked, neither would work. I installed the two new switches anyway and put the good old one up on the shelf to keep as a spare. It was pretty late by the time I got home. But at least the network was back up and going by the time everyone went in the next morning.
Actually, I must admit, that wasn't really the worst day of my user support life -- far from it. But it was the worst since my last one. What about you? Tell us about your worst user support day or maybe just the latest one.