Security

Manage client server OS patching with these best practices

Follow these best practices to ensure the server OS patch process runs smoothly and doesn't introduce new issues and possibly sour the client relationship.

Many IT consultancies develop managed services packages that ensure clients' servers are properly updated as part of a larger monthly maintenance contract. Other consultants are simply responsible for ensuring clients' servers remain current with the latest OS patches and updates. Either way, regularly downloading and installing Apple, Linux, and Microsoft security patches, performance updates, and other OS hotfixes doesn't sound like that big of a deal, but seasoned veterans know that installing the wrong OS updates at the wrong time can cost you a client. Here's how to keep OS patching from souring a client relationship.

Determine upfront how quickly server updates will be deployed

If a client demands that your IT consultancy download and install OS performance patches and updates immediately upon release, you should educate them that new patches and updates frequently introduce other issues. For example, a security patch could: create a conflict with critical financial management software (such as .NET Framework patches recently did for QuickBooks users), prompt antivirus software to fail or freeze (as my office suspects occurred recently on some machines), or adversely impact a server's performance. Consultants should ideally have time to test the patches in a non-production environment, or at least to learn what issues other business and organizations are experiencing.

If the consultancy isn't given proper time to monitor issues that arise for other first adopters, the client must understand incompatibilities, unplanned outages, and performance failures could occur. Unfortunately, those are the possible consequences for loading OS patches as soon as they're released.

Agree how OS patches will be tested

If a client requests that your consultancy test OS updates prior to installation, you need to ensure that both parties agree about what constitutes testing. If the client operates two Windows 2008 servers (one a domain controller with proprietary software, and a second providing terminal services functionality), the client must understand they must purchase and maintain a second redundant network if they intend for your office to fully test patch compatibility and performance prior to deployment within the production environment.

Due to cost, many clients will likely request that you delay installing the patch for a few weeks and review first adopters' experiences instead. Even within 10 days of the patch's release date, you can do some quick Google searches to see whether widespread or even isolated issues are occurring due to the update.

Specify when server reboots will occur

Operating systems frequently require rebooting after the installation of performance patches, security updates, and other OS hotfixes. I've been unlucky in guessing when clients would prefer for the server reboots to occur. Don't make the same mistake.

I've discovered that some clients work on Sunday afternoons, while others catch up after-hours (as late as two or three in the morning) on weekdays, so there's no hard and fast rule as to when it's safe to reboot a server. I recommend making sure the client understands that a specific day and time are fair game for regularly rebooting the server. With such an agreement in place, it shouldn't be necessary to work through complex calling trees, all-company emails, and other arcane methods of ensuring all users understand that a server will be rebooted every time it requires restarting. Whether it's the third Friday of every month at 6:00 P.M. or every Monday at 6:00 A.M., ensure clients understand servers need to be rebooted and that, to minimize communication and other issues, the same reboot windows can be set each month.

Nothing easy about it

The only individuals I hear saying it's no big deal to patch a server OS typically administer servers for a single business. Most IT consultants support dozens if not hundreds of servers for numerous organizations, each with different schedules, needs, and requirements. To ensure the server OS patch process runs smoothly and doesn't introduce its own incompatibilities, frustrations, or other issues, work through these steps with all clients.

About

Erik Eckel owns and operates two technology companies. As a managing partner with Louisville Geek, he works daily as an IT consultant to assist small businesses in overcoming technology challenges and maximizing IT investments. He is also president o...

6 comments
xeox
xeox

We had exactly the same problem as you do (not only with patching running machines, but also with setting new delivered machines to the latest patch level). We use scriptting and the command line WuInstall and in combination with psexec (see http://www.wuinstall.com for more information), and it works quite fine, not much scripting except settings certain WuInstall parameters needs to be done, it comes with a free basic version and a paid version with more options. As soon as no more updats are available, WuInstall gives you a return code signalling this and the machine can be considered patched. Works very well in combination with WSUS for pre-selecting updates.

Justin James
Justin James

My big pain point is actually doing the patching itself. I need to oversee the process and ensure that it doesn't happen the moment a patch is released (I use WSUS, so I have control over that). At the same time, I want to be able to kick it off manually, because I need to test things after the patching is done. In other words, how do I simply say, "all servers, patch and reboot now"? Right now, I'm going to each server individually, getting the patches I release via WSUS, installing them, rebooting them... it's a mess. I have a giant spreadsheet of all of the servers and the status they are at in the process, and juggling it is a nightmare. J.Ja

Justin James
Justin James

I use the spreadsheet to track the following: * Updates installing * Machine rebooted * Any more updates to install? * All services up and running? * Any issues found post-patching? * Complete? I would love to be able to just blast the patches out, know that X hours later all of the patching is 100% done (sometimes installing a patch and rebooting then causes another patch to have to be installed) and servers are rebooted, and that I can go in and test. Setting the servers to patch at a certain time does not work out well for me, because sometimes I need to adjust that time due to my schedule. J.Ja

Justin James
Justin James

Right now, interactively patching the 15 VMs I have (plus a few physical machines) takes up a few hours of time. What do people in a bigger shop do? That's what I'm looking for. I can't imagine that they do what I'm doing, which is going to each machine, triggering the patch installs, and then rebooting, then logging in, checking for more patches, etc. A company like a hosting company who has thousands of machines would go nuts. Is that *really* what people are doing? If not, is there a better way? For example, do people use a script to force the installs? Is there a System Center application that can let me select a machine and tell it to patch and monitor that from a central console? J.Ja

junior_gong
junior_gong

Justin - I don't think that BLASTING out patches is a good idea. I think it is best to be interactively involved in the act of patching to some extent. Before installing a set of patches you should see in WSUS that there are x amount of patches to install in total. If your server installs x amount of patches and requires a reboot before the balance can be installed - this will be reflected in WSUS once the machine checks back in. You can run "wuauclt.exe /detectnow" to have the machine check in instantly depending what the interval is set to. All in all... I think "set it and forget it" patching is what you are asking for. This is very unrealistic based on the tools available and the complexity of patching windows...

Editor's Picks