Discussion on:

The worst day of my user support life

Re: the original blog piece:

http://blogs.techrepublic.com.com/helpdesk/?p=330

Tell us about your worst user support day, or maybe just the latest one.

Three hours to diagnose a bad switch, and no backup hardware? Hmm?.

Three hours to diagnose a bad switch, and only one switch was bad out of two. Seems like someone needs to revisit their fail-over strategies. If there were two, why didn't the second (good switch) take over. Fix the process, or hire a new support professional.

Jeez...you two are horrible...like you have never had to spend hours on something that you couldn't figure out. You guys think you are absolutely so perfect and have all the answers. Gimme a break!

It's not called being perfect. It is called contingency planning. If you worked for a company and had situations like this, you would understand. But since you have never dealth with implementing a working backup process/strategy for your company's infrastructure, you are limited to foreseeing what could possibly happen in the event of a disaster.

I m not sure whether or not you have redundancy link between your switches. Next time make sure you have more than one link between your switches and router in case one of the port in either switch go bad.

I think the point of the story was completely missed by some of the "experts" responding to the post. The point of the story was about your worse day - not disaster recovery strategies. Although valuable input, I don't believe "WOW" is really an appropriate response. I am sure that if I examined some of your networks, I could find a few "WOW" factors spread around your IT infrastructure.

As an IT professional, I am not very quick to judge the efforts of those previously before me. My job is to fix it, and move forward - learning from my mistakes and other's mistakes. To those who hide in IT closets, behind their computer terminals, or their own perceived technical expertise, this is for you: a real professional understands that IT solutions are a combination of resources available, staff, money, scheduling, people, etc. - in addition to the technical know how.

The story started off by saying that he was a sole IT guy, they were a small company, and implied that he wasn't even doing that full time (because he had time to work on a design). Maybe if some of you took the time to sharpen your listening skills, you would take your "technical" skills to a whole new level. Computer room "tech heads only" who define their self-worth by the length of their IT vocabulary and wizardry, are a dime a dozen. IT professionals with interpersonal and business analytical skills are a real asset and hard to find.

I couldn't have said it better, Dave. As an IT professional of 21 years, I have seen my share of "experts" who excel at tearing someone down to make themselves look better. It is the rare gem who has excellent interpersonal communication talent on top of excellent diagnostic and technical expertise skills.

I've been in IT for 32 years and have seen many experts come and go, most of them no longer in this field of work. Never judge someone else's work, just keep learning and worry about your own. People who live in a glass house should not throw stones.

I don't claim to be an IT pro, I just have to take care of our home office network of three computers and a few printers. But I can totally agree with the idea of a "professional" being more than just the tech know-how. Thanks!

Just yesterday I did a dumb thing that made for some extra work. I had decided to tweak the BIOS in my personal machine, and totally forgot that I was using the onboard video card. In the process I disabled the VGA cache since (I thought) I wasn't using it, and then remembered after I rebooted and heard the tell-tale three beep pattern that I was indeed using the onboard video. Temporarily installing another video card and resetting the cache resolved the issue, but it was the dumbest mistake I've made (in a while, at least) and made for my own self-inflicted worst day!

I appreciate that.

The sole reason this was one of his worst days was because he obviously didn't know how to troubleshoot a network issue correctly.

The use of simple tools like ping, tracert or even wireshark could have resolved this issue in a lot less time.

Being the sole IT guy is not an excuse. I?m currently the only systems administrator at my company and when they hired me they expected me to be qualified.

I don?t claim to be an ?expert?, but I have been in this industry for over 20 years. It just Wow?s me when I see unqualified people in charge of a company?s IT department.

http://techrepublic.com.com/5208-6230-0.html?forumID=102&threadID=281440&messageID=2664729

In '20 years in this industry' Mike you haven't learned to comprehend what you read and it's obvious you have limited social skills. Please climb back in to your closet and leave the discussion to the true professionals. Thanks.

You are correct sir; I didn?t comprehend a thing he wrote.

?At about 4:50 one of my users came up and said he lost his e-mail and Internet.?

?It was now close to 8:00, and the only place I could get a couple of switches was at the Micro Center store way across town, which closed at 9:00.?

Then he says in his last post:

?I skipped a lot of the details (perhaps I shouldn't have), I assumed it would be taken as it was intended (perhaps I shouldn't have), and I embellished and exaggerated a bit for effect (perhaps I shouldn't have).?

So, who didn?t comprehend?

The next time you ?true professionals? want to make crap up just to blog, don?t complain when people can?t see through the BS!

That sentence really hooked me. I need to adopt that attitude in many areas of my life. Thanks for a great response.

Dave, thanks for bringing some sanity back to what was otherwise a bit of a pie-throwing contest. Your response hit the nail on the head, and then some, plus tax. Clearly you 'talk it' AND you 'walk it'.

As for the original post, sounds like a story involving a problem, a diagnosis, a fix, and some extra preparation to avoid the same thing happening again in future. On top of that, there was little or no downtime, and the office staff were kept at ease.

All of the above sounds like highly professional behaviour to me. Plus, Joe, you had the balls to write about it and accept responses from the community, even if some of the members like to "scorn the base degrees by which they did ascend" (to quote Shakespeare, sort-of). Respect to you, Joe.

Well said.

A number of years ago, I was chosen to be the systems guy in a centralization project that brought 3 locations' IBM System 36s into a single AS/400 System at my location. (I became the only systems person for a Western Canadian company in the entire province of Saskatchewan as a result of this project.) At any rate, to house the new system we built a hermetically-sealed computer room complete with halon fire protection (I did say a number of years ago!) and the whole deal!
So, conversion weekend happens and I have backup tapes from 5 data sources (the 3 branches that were being merged and 2 satellite warehouses) that need to be restored and merged into a single data store. I had brought my sleeping bag and had prepared for the entire weekend, knowing that I would be babysitting and monitoring a whole string of processes to get everything together for Sunday-night shipping (distribution warehouse environment). I restored and manipulated the first 3 data sets without incident, and started the fourth (one of the largest) determining that it would likely take about 7 hours to do the restore. I considered actually going home for a real rest in my bed, but got to the door and we were having a good pouring rain outside, so decided to stay and read a good book instead, periodically checking the status display on the front panel of the AS/400 and monitoring the job queue (I was using multi-stream processors and segmented DASD to restore and manipulate files on the go to shorten the overall process . . . turned out to be a mistake in the long run!).
About 4 hours into the restore, on one of my trips into the computer room, I noticed a drip starting from the ceiling. I started praying for the rain to end and began monitoring the situation with the drip as much as the restore process! Within about 10 minutes, I realized that the ceiling was starting to sag, right over the substantially expensive B45 model AS/400, brand new and totally my responsibility! I rolled the machine into one corner of the room, then another, but realized that I could not get it totally out of harm's way! As I was trying to position it in a 'safe' location. The ceiling began to crack and a spout of water began running onto the top of the cabinet. I have never hit a power switch so fast in my whole life! I grabbed a huge produce garbage bin from the warehouse, and a screwdriver from my IT toolkit and punctured the sagging ceiling in a controlled location to drain the water into the bin, protecting my equipment from being in the middle of a flood! It worked, and I soon had the incoming rain channeled away from my As/400. Alas! When I powered up again, the fact that I had been manipulating data as I restored to the data store caused the necessary power interruption to totally scramble data. All of the fancy technology of the new machine (redundancy down to the bus level, multi-streaming processors and data stores, etc. etc.)had not protected me from a roofing company failing to seal the top of a vent pipe from an old janitor's closet where my new computer room was now located! It didn't turn out to be just a bad day . . . I worked a 138-hour work week the next 7 days, first trying to sort out the mess in our data, then eventually coordinating having 2 other companies with AS/400 systems do our daily processing for us so we could continue operating, as well as assisting IBM field techs in wiping clean the entire system, and then going through the weekend long restore and merge plan again the following weekend, when I was a LOT more tired during those long waits for the processes to run. And the week after was a 90-hour week as I did my usual stuff, got my operators used to the new AS/400 environment and coordinated getting our data backed up and restored from the other 2 local AS/400 locations and supervising the removal of our data from their machines to ensure security had been handled (not that a hospital and a Cat manufacturing plant were likely to use a food distribution warehouse's data for their own purposes, but due diligence is due diligence, even when you are wiped to the bone!)

Just had to say "Hi"..

not very often I spot a fellow saskatchewonian in here! (or anywhere for that matter)

I hear its good an cold there the past few nights!

Nothing like a three week day to make wonder if a career move wouldn't be such a bad idea.

That is the definition of a really bad day!

We had a *small* leak in out server room roof for a year or so, but it was manageable because it wasn't affecting anything vital, until we too experienced "the bulge" when we had a downpour. I feel your pain, although I never had to work 100 hour weeks to resolve it!
Now a new roof has since been installed and all the gaps filled properly - so now I sleep better at night. I hope you can catch up on some much needed too!

We had an IT open office area directly under the labor and delivery department. They had a "husbands shower" where the labor coach could take a shower and clean up if it was a long labor. It wasn't used much, so about 2 months after it was added in the remodel, water started dripping through the ceiling. OOps, the contractor forgot to seal everything. We turned in a work order, and it came back as fixed, yet everytime that shower was used, we got wet. I kept big plastic sheets under my desk, and knew as soon as I heard, "tap, tap, tap" to bring out the plastic because I only had a few minutes until the water made it through the ceiling directly onto all of our equipment.

It didn't take me 3 hours to diagnose a bad switch. My total diagnostic time was probably 30 minutes or so. I just didn't think it was relevant to bore the readers with the details of each and every little thing I did between 6:00 and 8:00, much of which was out of the office doing some personal things.

If you're interested, these things included: attending an already scheduled Eagle Scout Board of Review for a young man in our Boy Scout troop (my planned dinner was to happen after that), I picked up some Chinese food on the way back to the office, I stopped for gas, I called my son and talked with him for a bit about the college applications he was filling out, and....... well, you get the idea.

As to not having backup hardware, that is a good point. Some companies, however, especially smaller ones like mine, simply don't have the budget to have a backup for each and every piece of hardware that might fail. Moreover, whether or not it's even a practical thing to do would depend on the dynamics and structure of the company itself. Every company's different, and such things are best determined on an individual basis. You seem to paint with too wide a brush.

By the way, we only have one chance to make a first impression. For your first ever post at TR, it's too bad it had such a negative tone.

"By the way, we only have one chance to make a first impression. For your first ever post at TR, it's too bad it had such a negative tone."

Agreed! What an ass.

As I explained to Joe_R, it was not lashing out at him, but merely giving a slap on the hand "shame on you" type of reply. I know a thing or two about this industry, and I know a heck of a lot about redundancy. First impressions are good, only if I'm looking to make friends, or looking for a job, in which case I am not trying to do either. As for your first impression, I don't think it is any better than mine, considering the fact that you have to curse in your reply. And to top that off, you are an IT Manager. Is that what your title displays? Righttt. I am also a manager, but I am definately hands on and aware of what's going on in the industry. I usually do not reply to these post, but look at them. I find them interesting. I just found the need to reply to this one after 4 years of being a member. How is this for negative, Mr. Curse man.

I enjoy reading this forum, and like many forums, I see posts from individuals like yourself who appear they need to show they are better then everyone else, know everything there is to know and never make mistakes. Maybe I offended your delicate nature by calling you an ass, but that was my first impression. Sorry. If you wanted to help the situation, there is a better way then slapping someone in the face and pointing out how superior you are. A simple suggestion would have been better.

I, like Joe_R, work for a limited budget IT department and sometimes we don't have the luxury of all the latest software and hardware on the shelf. It sounds to me like he did a great job of getting things back in order with what he had available.

What I find amusing is that "you've read forums about others like me who think we are better than everyone else". Well, the fact of the matter is, people like you label us that way. Never have I, or "we" mentioned that we are better than anyone. We just have more of an understanding due to experience of actual events, and we are vionaries. There is so much one can do on a limited budget. You do not need the latest software and hardware on the shelves. That is why "if you are a manager" you hire people who can get the job done, no matter the obstacle. I have guys at my firm that make a computer fly if I asked. Don't get mad at us because we are good at what we do. Yes, there are different ways things can be said, but I am not you, and our thoughts and processes to deliver are not the same. And for the record, maybe you should read carefully instead of twisting words. I said slap on the hand, not face. Never did I write I was better than anyone.

and you came across as a jack @ss

everyone lives and learns..and this was a post about "what a day..share your bad days"..not "tell me how stupid i am"

>>Three hours to diagnose a bad switch, and only one switch was bad out of two. Seems like someone needs to revisit their fail-over strategies. If there were two, why didn't the second (good switch) take over. Fix the process, or hire a new support professional.

This is not a "shame on you" slap on the hand, this is a "what an idiot you are" slap in the face. No, you didn't directly say, "I'm better than you are." But it is certainly implied. The attitude of this first post is further reflected in your responses to posts criticizing it.

You claim to be a "hands on" manager with years of experience in the field, which should include how to deal with people. If your posts in this thread are indicative of your attitude and I worked for you, I wouldn't make computers fly for you, I'd be looking for a job somewhere else.

Put your ego back in your pocket and lose the attitude, or go somewhere else. People with attitudes like yours have no place here.

When another member called IT Pro Doc an ass, this was part of the reply:

>>...considering the fact that you have to curse in your reply... How is this for negative, Mr. Curse man.

When I posted the message above this one (Lose the Tood), this was IT Pro Doc's reply:

---
From: IT Pro Doc
Subject: Ego
Message:
It's not an ego looser. It's called years of experience. I've dealt with loosers such as yourself who may have CCNA, CCNP, MCSE, MCSA, but you don't really know . Learn from experience is what I emplied. Get a life, and get your head out of those out in California. Maybe one day you'd get a promotion rather than being a helpdesk receptionist. Add this to the post "Looser."
---

Yes, I redacted the offensive curse words. And at IT Pro Doc's request, I'm adding this to the post.

By the way, the other poster was right.

IT Pro Doc, you're an ass.

Petty, so petty.
You are an IT manager? I'd hate to be your employee.

Hey Mike, I run my own company. My business model is basically one of a roaming IT dept for small businesses that do not have the budget for a full time IT pro. Trying to convince a small business owner to spend the money on hardware that they may or may not need in the future can be a very tough job until it actually happens. I do carry spares myself, but I try to keep that to a minimum because it is my money that is tied up until one of my clients has the need for it. I would agree that three hours to troubleshoot a bad switch might be a little excessive, but there is an old saying that kind of applies here, "there are those who have and those who will". Sometimes you run into situations that kick your butt, it is called learning from your mistakes or just plain experience. It really does not matter how qulified you are. I thought it was a good story just the same and I am sure that Joe R learned a lot from this experience.

We had a VP who had just got his first laptop, this was back in the mid 90's. One day he leaves for work, puts his laptop on top of the car, gets in and drives to work.

Needless to say when he got to work, no laptop, drove home and it was gone (this was a time when a decent laptop was very expensive).

They scrambled to get him a new one (being a VP and all). After he got it, left for work one morning, put his coffee and laptop on top of the car, got in and started driving to work. Realizing he didnt have his coffee he reached up to grab it. At that point he realized his laptop had slid off the car so he backed up to get it and ran over it.

A few weeks later he was let go...

Is always professional and good to have a recovery plan.

I believe will be good to have spare parts and hardware for emergencies (ups, routers, modems, switches, hard drives, etc). Even backup servers.

I believe you need redundant link connections between your switches and snmp monitoring strategy in place to receive alert notifications and avoid long troubleshooting hours.

I know what an 1 person IT department is... you need to dial with everything and when disaster appears, the stress bar go to the top.

In the last place i used to work for we had POE VOIP phones. One day one of the phones in the finance department stop working, the end user without ring I.T got a power plug and plugged the phone into the main power socket while it was running over POE needless to say the hole network came crashing down, the port messed up on the switch and the routing table 5 hours to find the problem and fix it so funny now when i look back at it.

My customer had a bad hard drive and I have to sent it away. I b/u the data and reformat the drive for security reasons. Then the customer phoned me and told me not everything had been back up. Fortunally, I bought an un-format utility and got his data back. Whew!

During a failure of office central air, several computers were overheating, partly due to too much excess junk stored on or near (or under or behind). I suggested moving them near the windows to allow better ambient cooling. One user left a computer by the window during a driving rain storm later in the day (LOUD BZZZZT!), another left it by the window and left it there with the window open after closing. Needless to say we were shy one system the next morning. I was reprimanded for not "adequately explaining aspects of my suggestion". WHat, don't leave electrical equipment exposed to water or don't leave expensive equipment in an open window when no one is home? ???????????

But it was far worse as a Death was involved and I had to go to a Coroners Court to explain why I had not made things safe. Seems that Fencing off the area of the Frames under Service with Temporary Fencing and removing the Power Plugs isn't sufficient Safety procedures to follow. The person who got killed had cut down the Temp Fencing and fitted new Plugs that I didn't know existed but it was my responsibility because I was a Professional and he was just a worker there who shouldn't have been in the building at the time let alone fitting new Power Plugs.

Funny thing was he wasn't even messing with things he shouldn't have been he was caring a tray of Coffee Cups back to the Data Entry Area when he steeped into one of the trenches with the covers removed and fell into one of the units that he had powered up and got electrocuted.

Even since that time I never trust anyone to do what they are supposed to or have any brains at all. Making assumptions that people know better will get you into a world of Hurt Every Time.

Col

When I think back to any day(s) that may have been abnormal which caused me to pull my hair out, I realized something, It never seems to be the technology that makes the day stand out...its the users responses and actions that made the day standout as a bad one. Besides if we didn't all love figuring out figuring out why things work that way and the challenge of figuring it out we wouldn't be in the IT field.

Well the story starts with being a small company that got purchased by a bigger company (this company owns the 3.x network) and I was in the middle of training a handful of people that would be taking over my role of desktop support, "freeing me up to work one more challenging things".

In the middle of this training, the CEO of the company stormed into the office swearing and tearing at me and accusing me of bring down the network and phones during a VERY important meeting with this new company. I would post what was said, but it would get blocked by most vulgarity filters. The new desktop people where all very surprised all of their jaws had dropped to the floor and they all just stood their in shock and amazement of what they just had witnessed.

Well needless to say this new issue jump to the top of the priority queue. I started looking at all of the obvious things like the firewall, router, etc. I found the issue when I went to the TelCo closet and saw that the Ts for phone and data where not communicating.

So being the due diligent IT guy I thought I would poke my head in to the CEOs office and let him know I found the issue and would let him know as soon as I knew when I could get him an ETA on it being operational again. Well that was a mistake, he ended up threatening to kick my ass and shove his foot into a location where the sun does not shine unless it was repaired in the next 30 min.

So I headed off to call our ISP and start getting a handle on this, which this in itself produce an issue because we had lost phone service and because of the building structure and electrical interference my cell phone did not work in or around the telco closet or the server room.

While I was on the cell phone with our ISP standing outside of the building just beginning to troubleshoot the issue when the CEO came storming out of his office straight toward me screaming about dealing with important issue and making personal phone calls during a business emergency. This person grabbed my cell phone from my hands and threw it across the parking lot, my poor little razor didn't make it through the impact. He then screamed at me to fix things and stormed off.

Well by this point I was more frustrated about how I was being treated than fixing the issue, but swallowing my pride I went back into the office and got another cell phone and called the ISP back.

After all that, when they finally rolled a truck out, the technician informed me that when he arrived the central point where all of out data and phone lines entered the office park was struck by a person in a truck and the entire office park was down.

Well, after all that, I worked with the repair technicians that came out to fix the issue to make sure all of our items where fully operational before they left. Which I was surprised it only took 4 hours for them to completely re-build it and test everyone's connections.

That I can say was one of the worst days I ever had supporting a network for a company. And so everyone knows, that CEO didn't last much longer under the new company. Larger companies usually have procedures in-place for dealing with issues like these, which is one of the only good things about working for a large company.

Man I am sorry you had to deal with this guy.

I don't take that kind of treatment well. Nice to see that you prevailed.

...and possibly carted off to jail, following my likely response to that sort of treatment.

...I would have simply left that guy with a dead network for his recently-trained desktop people to fix.

Yeah I really thought about just walking out or decking him and then walking out....but I am one of those people that tries not to leave one job without having another lined up.

Going through start up companies layoff after the bubble burst tends to make people a little more wary about having a job, and an un-fortunate side effect is I tend to put up with more ****.

But yes, justice is a dish best served cold. He was told that his services where no longer needed by when the company started showing up at our new facility (company consolidation / re-location) the name on his door had been changed....I just smiled knowing he felt about as ****** as I did that day.

Now that's more like it.....that crappy CEO ain't worth going to jail for...Lol

Yours is a case of pure Drama.....No one should have to deal with crap like that....I'm only glad that jerk of a CEO did not last!

Even though I'm not a regular contributor to TR, I humbly suggest that we all choose to ignore Shadowcast's pathetic posts. If ignored, like all trolls, he will find another site to bother.