Leadership optimize

Video: Five server upgrade mistakes IT professionals should avoid

Bill Detwiler highlights five mistakes IT professionals should always avoid when upgrading server hardware or software.

If you make a mistake when upgrading a user's desktop or laptop, many times you're only affecting that single user or perhaps a handful of people that depend on that user. But if you make a mistake when upgrading a server, you could stop dozens, hundreds, even thousand of users dead in their tracks. During this TR Dojo episode, I'll go over five mistakes you should always avoid when upgrading a server.

For those who prefer text to video, you can click the Transcript link that appears below the video player window or read Erik Eckel's article, "10 things to remember when upgrading servers," on which this video is based.

You can also sign up to receive the latest TR Dojo lessons through one or more of the following methods:

About

Bill Detwiler is Managing Editor of TechRepublic and Tech Pro Research and the host of Cracking Open, CNET and TechRepublic's popular online show. Prior to joining TechRepublic in 2000, Bill was an IT manager, database administrator, and desktop supp...

22 comments
smizoe
smizoe

I once restarted a server--remotely--after a routine maintenance and patch update. I waited for the server to come back online, which usually took 20 minutes. I waited 30 minutes: not up yet. I waited an hour: no luck. Next morning: still off-line. I went to the office (on a Saturday) and sure enough, the server was stuck at the boot-up POST screen. I thought "crap; there could be any number of reasons why the server's stuck" when it suddenly, for no apparent reason, began normal boot-up procedure. It turns out there was another server's keyboard resting on top of a few keys of its own keyboard, pressing them down. Once the obstruction was removed, the server started normally. Needless to say, I laughed hard and promptly felt embarrassed at my own negligence! Ever since, I try NOT to restart servers remotely if it can be avoided. And keyboard and mouse? I make sure they are free of any obstruction.

jrussell_75423
jrussell_75423

Taking precautions prior to changing things (h/w, s/w, or etc.) is always prudent. A supervisor who hasn't the time to do things properly isn't likely to have the job long.

lnchome
lnchome

Yes follow a supervisor's instructions not to take a precaution you think is necessary. this way when it gose south he will have to think, maybe is should of done what the tech. wanted to do first.

wcallahan
wcallahan

I have never come up against that, but my first reaction is no. Not without having a discussion with my supervisor about why. There are times when a supervisor has foreknowledge of circumstances that may have prompted the request. If the supervisor is just being reckless, it's time to take the discussion to that supervisor?s supervisor.

darkstate
darkstate

Well I'm hoping the Boss has done his due diligence and checked all the hardware works with the drivers etc, If so yes go right ahead- Thats why he is paid the big bucks right? If he hasn't checked all the boxes then no, Don't do anything without checking mr google first with your setup, but hey If you are mr no it all and things come unstuck really quickly after the upgrade,then It your fault and goodluck with the rest of you quite day.

rick
rick

I worked for a consulting company that provided tech support to a company. Company X had a network tech but when he got in over his head he called us. We also hosted their e-commerce website. We get a call about 11:00 AM on a Thursday that the website is down. After some time of troubleshooting and my repeated question of "has anything changed" he mentions in passing "well Bill put service pack 4 on the server a little bit ago but what could that have done?" They upgraded W2K from SP3 to SP4 on a production server in the middle of a production day and it didn't occur to them for the first 30 minutes of my phone call that it would constitute a major change!! So kids, I have a teachable moment here: don't perform major upgrades on production servers in the middle of a production day :)

dele1jim
dele1jim

My big problem recently when upgrading servers is not knowing all the other systems that will be impacted. For Example, I just upgraded our Exchange server from 2003 to 2010. After I had the server built and started moving users, I realized that our current backup solution (Backup Exec 12) could not backup a 64-bit Exchange server on the current 32-bit O/S. So now I a scrambling to setup a new 64-bit Backup Exec 2010 server, instead of moving user accounts to the new Exchange server as I had hoped.

Gilbertr14
Gilbertr14

No, but if he insists, document, or have him verify the instruction via email.

skodz
skodz

Your last backup is only as good as your last restore. Not only should you make sure the backup worked, it is worth performing a test restore before a server upgrade, so you have a known good backup to roll back with, if required.

Migration Expert Zone
Migration Expert Zone

To this list I would add, "Don't panic." When something goes wrong, IT types often make the mistake of getting so caught up in the situation that they make matters worse. Best bet: step away from the server, go take a walk, and think through the problem.

Bill Detwiler
Bill Detwiler

In the above TR Dojo post, I show highlight five mistakes IT professionals should always avoid when upgrading server hardware or software. Making a data backup and even a system image top our list. Unfortunately, some TechRepublic members have reported that upper management has shot down their plans to image a machine before an upgrade. Jmbrasfield described such a situation: "As to the imaging, I've recommended it every time, and every time I've been shot down by higher ups claiming it takes to much time. I was actually told by one that I had 15 min. to shut it down, replace the component, and get it back on line." Should you follow a supervisor's instructions not to take a precaution you think is necessary? Take the poll in the above post and let me know. And, let me know if such a situation ever ended badly. Did the manager take responsibility when things went wrong? Were you left holding the bag? Original post and poll: http://blogs.techrepublic.com.com/itdojo/?p=1732 Jmbrasfield's post: http://techrepublic.com.com/5208-13583-0.html?forumID=102&threadID=328086&messageID=3267368

gpott
gpott

A big yes to this one. IT types worth their salt are very good at quickly predicting the cascade of events and impacts of various situations. While this is valuable in pre planning, it can result in panic and an avalanche of thoughts of doom if something bad happens NOW. I'm sure everyone has had that hot flush of dread that immediately follows the realisation that I've done something / something's gone wrong. Get up, walk away until you can think straight then come back with the solution and fix the existing problem, dismiss the problems you have conjured up. These situations don't happen very often but when they do they are burned forever, PTSD-like, into your memory.

hauskins
hauskins

If you have plenty of time in advance, you can start an image by using "rsync" or a similar package to create an image. Timing is the key, start an image creation process a few days in advance and then keep it updated (sync file changed) so that when you are ready to make the big jump to do upgrade work, you do one last sync, which should only take a few minutes. That is what I do on many systems that I upgrade as far as the OS and even applications goes.

TGGIII
TGGIII

The assumption is that your supervisor has the postional authority to order you to do the upgrade, "on the fly." My approach is that whenever I am being asked to do something that is suboptimal (not immoral) is to confirm the directive in writing. If the supervisor has a defensable reason, you are both covered; if there is an audit after the crash, you are covered.

BlazNT2
BlazNT2

First off this was something I was called in to fix after the Exchange server was down for 4 days. I think the guy before me made all 5 mistakes. When I got there to baby sit the server so the IT professional could sleep for a couple hours. Yes my company just wanted me to babysit the server while he got some sleep. When I arrived and looked over the server with the IT professional I quickly determined through his own words the he does not do hardware just software. No recent backup (30 days old). He had never opened the server. This was one of those install the update and reboot the server. It never came back up. I determined it was the RAID controller after 15 min and ordered one from IBM. 4 hours later it showed up. Not the correct part but a replacement part. Ordered a new/original RAID controller. 5 hours later it came in. I put it in the server and the upgrade continued. All was well luckily. Not by planning though. The best part is that the IT professional returned just in time for the fix so he could take all the credit. You just have to love your job.

sub_techrep
sub_techrep

If your supervisor does NOT have an IT background, make sure you explain the downside of skipping the step you think is essential. If your supervisor does have an IT background and tells you to skip a step that you think is essential, always ask "why?", just in case they have a good point and you could learn something. Either way, if you still disagree with them and they persist in their design, send them an email which says "Please confirm that you do not want me to before I

techjmj
techjmj

And document document document...

PlexusSage
PlexusSage

You should always have a backout plan with plenty of time to roll back changes before everyone shows up for work. As I gear up for a domain and Exchange upgrade for a client, I'll make good use of trusty ShadowProtect CD. If the project goes south over the weekend, all will be the same as it was on Monday morning as it was on Friday afternoon.

Realvdude
Realvdude

Pretty much covered why I also said Yes. In the end, it all comes down to each situation. If you ignore them and take precautions, will you lose your job? If you follow their instructions and something goes wrong, will you loose your job? You've managed to put a positive spin on something that could be a negative experience.

mike
mike

I to answerd YES because you are the subordante. Now with that said of course it should be questioned and documented. Always cover your butt!.

asics447
asics447

Yes and document document and some more documentation. Happened to me many times in the past. You tell them what could and will go wrong, document it and then you have to fix it. Right wrong or in-different I do what I am told and CYA!!! and regardless of the outcome you will be blamed for it.

amado.puentes
amado.puentes

If skipping a procedure that you know will end up in a catastrophe, just because your supervisor decided that it was to be bypassed, can be morally criminal, totally irresponsible and outright unprofessional. Get chewed out for saving money time and customers, not for screwing up a job which ended in a catastrophe. Amado Puentes BCC New York