IT Employment

Ask something
Discussion Locked

When the lights go down

0 helpful
27 replies

by tig2 1 view

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

I have spent a significant part of my life managing after the lights go down on an install. That has been the time that I am allowed to get my team moving. We managed to miss the milkman and the postboy- leaving many of us to wonder about many things.

This all leaves me curious. What happens when the lights go down on your job or place of employment? Do you get called out? Are you expected to know things that you don't and, realistically, couldn't or shouldn't, know???

Tell all- I promise to never use your name in my upcoming book, "Stupid Sh*t I have Managed". I might not even bother to write the book.

Seriously- how do you manage expectations? How do you keep from getting lost in the "goo"? How do you keep your sanity in the face of the ridiculous?

Tiny minds- and this bouncy one- would love to know.

C'mon- you know you want to answer...

I hate myself but I am going to hit all the primary tags. If you have never been in a place where you had to ask all these questions, I apologize now.

27 replies

by gsg

First, Tigger, I was thinking the other day that I hadn't seen you around in a while. Good to see you!

The lights never go off here because we're a hospital and we're responsible to make sure systems from the system that manages the patient's diet and menu to the system that's in the cardiac surgery room, and the system that dispenses the patient's drugs are up and running without errors. After all, if the Best Buy systems go down people can't buy that gigantor flat panel. If our systems go down, people could die.

We have 3 people that take 24 x 7 calls. These are our tech support team, and these guys are amazing. We have over 50 medical systems, and quite a few non-medical systems, and they have to know a bit about each one. The system administrators have to be experts on anywhere from 2 to 8 systems. If one of the on-call guys gets a call, I'd say 98% of the time they can take care of it without calling us.

We have a system of primary sys admins and secondary sys admins, so that if the primary person is not available, the secondary is called. However, it does sometimes work out that neither are available, so when that happens, one of us who've been here a long time, or someone like me who touches almost every system in some way is called.

I manage an interface engine where I build interfaces so that the disparate medical systems can share data. Because of that, I have to know some odd info about the various systems. I'm not an expert on them, but I understand the data flow and some of the underlying structure. I can at least help identify the cause of a problem and give advice on what I think the underlying cause is.

The last level of support is the vendor. We pay for 24 x 7 support for all of our systems. If we have to, we call the vendor and make it a system critical call to get it fixed immediately.

We also keep what we call our top 10 lists. Each application has it's own web page on Share Point, and we list the top 10 known issues that crop up, and how to fix it.

by tig2

Lately, I have gotten an "up close and personal" look at how hospital systems work... and don't. From what I have seen, it isn't the IT guys that fall down on the job. Here, anyway, it is the health care workers themselves. My experience has been that the IT staff has been fabulous.

by gsg

Health care tech is rough. The gov't passes regulations without a thought to what it will mean to the hospitals in the cost to implement, time, risk, etc... then they have the nerve to say that they have to do something about the cost of healthcare.

We try very hard to train, support, hold hands, give tissues to the nurses, but we get blamed for "forcing" tech on them. One of the nicest, sweetest people I know was training some nurese last week, and after listening to a person gripe and stretch a short training session into a couple of hours because of a bad attitude, this sweet person snapped, and said, "This will be as hard as you want to make it, it's up to you." After that, training went smoothly.

Then, there's the other type of clinical person. The one that uses the tools we give them, uses them correctly, which makes the level of patient safety rise, etc... This is the nurse that does well with or without technology.

I've spent many, many hours in the hospital with my mother in the past 15 years, so I see it from the patient side, IT side, and the nurse side.

by mjd420nova

I have a number of clients that have signed 24/7 contracts that require someone be on call at all times. We rotate weekly and this sometimes interferes with those who have regular 9 to 5 commitments that must be met also. Comp time becomes a life saver and can often put undue strain on others but we all have to serve our time and meet those needs that are unpredictable. Weekends are just as busy as weekdays and those who get called out while relishing in family events know these needs are just part of the job.

by locolobo

It's been a while since I stayed late here. My boss wants me in first thing in case of problems. He's a late riser and just won't get in early except for major end-of-the-world problems. To him sunrise is a perfect end of the day. So I'm usually the first in nowadays.

A few years ago we had one of those crazy priority projects. Arrive at 9ish, stay until you can barely think anymore, come back the next day and do it again. It lasted for 6 months. What did we do after all the "children" went home? Condiment tossing contests, added audio files to application pages, played handball in the parking lot, and we did some work.

Usually it would be my boss who would instigate this. After he could see we were bogging down mentally, he would find something interesting to do, listen to, view, etc. It's amazing how a little fun can restore your thinking. The trick is to not let it take over your time.

by tig2

I was working for a major health insurance provider and we had to transition the network from something that was known (kinda) to something new. This required a physical touch to every single machine on the site and quite a bit of hair pulling- our own, of course.

My boss at the time was a great guy who wouldn't ask his team to do anything that he wasn't willing to do as well. This particular weekend was his birthday. His wife was not thrilled to have him work (WAAAAY) late, but brought us all dinner regardless.

We had done the first pass- every machine that we could touch and get a small piece of software installed on... that didn't balk- and had a good list working so we knew what we needed to do over the weekend. And then we found the "box from H*ll".

I worked on that sucker for a good hour. When I reached the four letter word stage, another team mate took over. He beat on that thing for thirty minutes before we called in the Boss.

Our Boss sat down with a list of things to do- all of which had been done by myself or my team mate. He was going to SOLVE this! I parked myself so that I could learn from the Master.

About 35 minutes later, I found myself in a new role- keep the Boss from throwing that *&^*% computer through the floor to ceiling windows in the workspace we were in. We sent him home to spend what little was left of his birthday with his wife.

Over the next two days, I spent many hours chasing stuff down and even "swimming" under the raised floor to find all of the connection points and document them. The Boss was there and working with us the whole time. We found plenty of things to laugh about and still managed to find plenty of time to do the job.

I love "Boss" stories like that. I still remember- with MUCH respect- that Boss.

Thanks Loco!

by tig2

You really are way too easy.

In your world, the lights going down is simply a hardware issue and not your problem. The SO is a software architect- I know these things.

by dmambo

...and I'm not referring to jd.

I AM the IT dept in my plant, so if there's a power surge, 2nd shift crisis, someone else working late causing a problem, etc., there's really nobody else to handle it. If I'm on vacation, I leave a primary contact, but there's only so much he/she can do remotely and without detailed knowledge of my site/environment, so I take calls anytime. Just part of the job. Sure beats supervising 80 line workers!!!

by tig2

I have never been in the role of the "only owner" except at home. I keep the network running here... with a heavy reliance on Comcast to provide the signal.

The fact that you are the primary "owner" of all that is IT must be an interesting torch to carry. Regardless, it sounds to me like you enjoy it.

So here's an odd off-shoot to my original question... What are a few things that you love about your job?

(((hugs))) Mambo. Missed you.

by dmambo

Well, defining "love" verrry loosely, here are a few:
- I have to learn at least a little bit about everything IT-related that goes on around here. Whether it's my initiative or a corporate mandate, there's no such thing as someone else's project.
- My boss is in a different state. Though he seems to be a really good guy, I've only met him face-to-face twice - and once was playing pool.
- Good coffee
- Sometimes I feel like I'm the smartest person in the factory, at least in my little area of expertise. A nice feeling.
- I can take a couple of minutes during the day to do things like this. Makes up for the nighttime ERP system updates a few times per month and the late phone calls.
- If I make a mistake, generally, nobody knows about it if I can fix it myself. Though I'll usually let my boss know anyway.

Nice to be back!

by jck

Now I have that Journey song in my head...

When the lights...go down...in the city...
And the sun shines on...the baaaaaaay...

:^0

by tig2

Believe it or not, I wasn't going there. But now that you mention it... I never liked that song...

Love you tho...

by jck

with the answer one of my heroes, James Taylor, did on his VH1 Storytellers episodes when a woman in the crowd said "I love you.":

"Um...can I just buy you a house or something?" :^0

BTW, I love Journey. I wish I could have seen them before Steve Perry was extricated. They are one of my 5 favorite musical acts ever.

Wow. Someone here loves me. How do I do a blush smiley? :)

by jdclyde

Every now and then, people I support need to be reminded what my job is and what my job isn't.

No, I do not know anything about your copy machine, nor your fax machine, but I can run the heck out of the coffee machine.

No, I do not know every application/program that they can install on their systems just because I am the IT guy. Hand me a blank check and a week to go get training on it and we will see.

Another thing I am quick to do is correct peoples misconceptions on "the dumb computer". Yeah, it is the computer that is dumb..... It should have known you didn't intend to click on that button, right?

"The network is down", when it is actually a single application that has hung. What have you tried, and what was the results. Oh, you got "an error"? Gotcha, so I will fix "a problem"?

by nicknielsen

After all, if they weren't, they would do what we want them to do and not what we tell them to do. :D

by dmambo

One of my favorite helpful tips when I'm troubleshooting a user's problem is when I'm told "There was an error message that said something, but I don't remember what."

by forum surfer

There are security officers posted all night so coming back when I am on call is uneventful. As a matter of fact things have been smooth as of late. I haven't been paged in 2 months for anything serious enough to require a hands on visit. The only time I really have any issues are when different IT departments that utilize our data and/or services get involved. These guys tend to wait until 5 minutes after closing to ask for help on a problem they've dealt with all day. Grrrrrr....

Although there have been a few occasions where the lights go out and a new coworker from a different department I'm dating helped me "clean off my desk." :)

My last job required me to visit branch banks. I once went into one not realizing this particular branch had a night watchman. Needless to say, we bumped into each other at 11 pm and scared each other silly. Luckily for me, he wasn't trigger happy with a fast draw.

by nicknielsen

I'm on call 24/7. Additionally, because I'm the only tech from my division in my area, I'm on call every day the stores are open. Since they're only closed on Christmas Day, I'm essentially on call 24/7/364. (There are several other techs in my area, but they are assigned to a different division; contract stipulations do not allow division crossover in either direction.)

In reality, I don't get that many late evening or early morning call-outs. Most outages severe enough to justify a page are usually discovered/reported during pharmacy open hours (0900-2100). I'm much more likely to have a long day than an evening call-out.

I am, however, the de facto SME in my division for several critical store systems, so I do receive calls from other techs at odd hours, even though I don't have to go anywhere.

The most irritating thing about the whole process is that up until spring 2008, I was receiving $40/day in on-call pay every weekend and holiday. When that was eliminated, we were told we would be alternating weeks on-call with other techs. The other techs in my division get the week-on/week-off. Because of my situation, I don't; I just took what was essentially a $2/hour pay cut.

by nexs

In my case, I'm at a power station... so if the power goes down, then we have a very serious problem!
In a worst case scenario, i don't get the callout. I'm a mere contractor haha! But the business i'm based at have an on-call staff which changes each week (i think) but if the S**t hits the fan then the higher level experts will be called upon... I'm rarely going to be the one to call as I look after the helpdesk.

by seanferd

and "should be"s that I have to either dispel or bypass entirely.

"Someone hacked into my remote and broke it." Translation: Power adapter cord is bad on the gateway.

by jck

It's hard for me to know. Plus, I use all the ones in MSN and Yahoo Messenger, so remembering an entire other set is confusing.

Thanks, Nick :)

by nicknielsen

with a shortcut on the quick links toolbar. And since Foxit Reader doesn't take but a couple of seconds to load, they're pretty much "always there." If you want a copy, PM me and I'll send you one.

This discussion is closed to new replies.