Security

How should you handle software updates?

Whether you're talking about computers in a well-secured enterprise network or those plugged directly into a DSL modem at home affects how software patching should be handled.

Different people tend to advocate different approaches to software updating. The single most common approach I have seen recommended is very simple -- even simplistic: keep your system updated with all recent patches. It is suggested in the statements of people who want to pass themselves off as security experts (or at least security aware) all the time. A reasonable assumption, if you take that advice, would be that you should have Automatic Updates running on your Microsoft Windows systems so that every Patch Tuesday you get updated with the newest security patches.

The world is not so simple, though. For example, Microsoft issued a patch for a vulnerability in its SQL Server software in late 2002. Later, two unrelated patches issued for Microsoft Windows effectively deactivated the preceding SQL vulnerability patch. In 2003, a bit of malicious mobile code that came to be known far and wide as the SQL Slammer worm crippled much of the Internet because of systems that had been fully patched, but had the relevant security patch deactivated by later patches. This was not a universal problem; it depended on the order in which certain patches were applied, but the number of computers it affected was mind-boggling nonetheless.

Microsoft issued press releases chastising system administrators for failing to keep their systems properly patched. People lost jobs over this. Ironically, the problem was not people who didn't keep their systems updated; it was people who did so too zealously, and too uncritically. This is exactly the sort of situation many system and network administrators fear when they turn off Windows Automatic Updates.

The preferred alternative on many well-secured, firewalled, gateway scanned networks with careful usage policies is to have automatic updating mechanisms turned off so that network administrators can test all updates on systems set aside for exactly that purpose, and ensure there will be no interruptions to daily operations when a given patch is distributed. Examples of annoyances that admins encounter while testing patches (or while letting patches get applied without testing, of course) have included not only patch conflicts such as the SQL vulnerability conflict in 2002/3, but also:

  • unwanted installs of Windows XP's Service Pack 2
  • unwanted installs of Windows Genuine Advantage and similar DRM measures
  • unwanted version updates of Internet Explorer
  • unwanted default configuration resets
  • unwanted system reboots
  • unwanted reactivations of Automatic Updates

The common wisdom among those who actually know about, and professionally deal with, this kind of problem is that one should never allow computers in one's charge to update software automatically. Updates should always be tested before being applied. It is a popular point of view among network administrators who have been in the business for a decade or so, but not among Microsoft employees like Bret Arsenault.

There are situations in which software updates should not be delayed, however. Such circumstances involve computers that will be connected more directly to the Internet or other hostile networks before there's a chance to test security patches. Such computers might be mobile systems, such as laptops that are taken on business trips, as well as non-mobile systems such as the gaming system you have running World of Warcraft on your desk at home without the resources to maintain a test network to determine the problems that might arise with dangerous software updates.

Sometimes, you might simply have to apply security patches -- or even functionality patches, if there's some functionality being rolled out that you can't pass up -- without taking the time to properly test the software updates first. I myself do not currently have much of a test network at home, and don't get to test patches as thoroughly as I would like before applying them to my laptop. I do, however, endeavor to avoid applying updates that I don't have to without advance testing. Luckily, my laptop runs FreeBSD, so I can use portaudit to determine what software updates are really critical, and ignore the rest until I get a chance to test them or otherwise make a less hurried decision about what to do with them.

This suggests another rule of thumb of software updating; that one need not apply all patches. Most software updates have nothing to do with fixing security vulnerabilities. As a result, most software updates never need to be applied unless there's some compelling need for the functionality changes the updates provide. It is unfortunate that many operating systems don't do a reasonable job of clearly differentiating between security updates and nonsecurity functionality updates, assuming that all users will want to apply all patches. In some extreme cases, failing to apply a given patch may in fact interfere with the ability to apply a different patch later, as is too often the case with MS Windows patches.

In cases where your computer is not well-secured against mobile malicious code and the specific attention of malicious security crackers -- where it lacks good external firewall and gateway scanning protection, for instance, since (one hopes) there will always be decent on-system firewall and scanning software -- it may even be necessary to apply almost all updates immediately, without any testing.

The canonical example is the home user with a Microsoft Windows desktop system plugged directly into a DSL transceiver (more commonly, and inaccurately, called a "DSL modem"). Under such circumstances, the threat represented by the direct, unfiltered connection to the ISP's network is marginally greater than that of an unwanted software update. The recent explosive growth of Conficker Worm infections is a perfect example of why such systems' security updates should be less critically examined if such scrutiny of patches might delay application of important vulnerability fixes. As physically carrying the worm from network to network has proven to be a big problem with Conficker, it's also an incidental example of the importance of good organizational security policy.

If the home user scenario describes you (even if you're one of the Three in 10 Windows PCs still vulnerable to Conficker exploit), however, you should not simply take it as justification for just turning on Windows Automatic Updates and feeling smug about security. This is not a good answer to the problem, though it may be an effectively necessary answer in some cases. It's just the lesser of two evils. The importance of maintaining good backups -- always very important -- is increased significantly by the automatic application of untested patches, for one thing.

For another, such a policy is only "good" by comparison with a policy that leaves the system unprotected against outside threats, as in the case of a computer that has no external firewall and scanning protection. Finally, on MS Windows systems in particular, there is rarely any way to only automate the application of security patches, and leave functionality patches unrelated to security to be applied "manually", which means that you are at even more significant risk of unwanted or faulty updates making hash of your system configuration than if you were only applying the security patches without testing.


About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

38 comments
jss
jss

Security updates are quickly tested and applied; others are tested thoroughly before being applied.

derek
derek

I agree with the automatic suggestion, however recently I noticed that with many of the recent patches from MS, I am having to change the MTU on our wireless routers and wap... I think there is a connection here as nothing else has changed (no firmware upgrades, just ms updates).

netman
netman

Depends on the systems that are used. All systems have AV installed on them with a technology provided by the AV provider that detects behaviour that could be virus like. Ensuring that hardware devices are also patched to newest firmware also helps.

Craig_B
Craig_B

First I research and try to understand what the latest patches are trying to do. Then I apply to an early adopter group who know the risks and know how to handle any issues. Next any public exposed machines get updated, then most servers and workstations. Lastly servers that need "special" attention. This has worked well for us.

sleferink
sleferink

Workstation patches are reviewed, quickly tested with critcial applications and analyzed for impact. Patches are rolled out through a network deployment based on the type of patch, need and impact. We normally delay rollout by a day to complete testing and make sure the patch is not rereleased (typically this happens in the first 24 hours). Occasionally we patch the same day if there is an actual virus running rampant and security newsgroups state there is a dire emergency to apply the patch immediately. All portable devices are set for automatic updates and if we are aware of an upcoming update that we do not want to go to the portable devices we will warn device users to run a script to prevent the download of the specific updates such as IE. If they don't get to it then they have to deal with the consequences. If an automatic patch wrecks havoc then we will set up an uninstall and have them connect to the network with their device to have it removed.

Neon Samurai
Neon Samurai

I'm currently reviewing the existing update policy so none of the options apply. For disinterested home users, I recomment "download but ask before applying" so you can sit on them a few days or at least see what is being added to your system. Risks like Slammer exist but making the attempt is more often better than nothing. For work machines, turn off all auto-updates and manage the process manually or from a management appliance after testing against a non-production machine and waiting a few days to see what turns up in the news. The latest unschedualled patch being an example. For my own machines, it's all manual updates too. I don't want my testing tools anouncing themselves on a client's network by looking for updates. I also want to research updates for the rest of my machines before they are applied. (Quicktime always wanting to include safari and iTunes being an excellent example of why to run it by hand).

wowpaladin09
wowpaladin09

Does this mean that patching World of Warcraft can potentially compromise my system? I'm a very careful gamer and I don't visit suspicious links and only opt for wow gold sites. I don't want to get keylogged or something.

Justin James
Justin James

I think one of the biggest issues here, is "how do I truly test a patch?" In the open source world, I supposed a small number of people inspect the source code (as well as any files related to the install of the patch, such as the makefile's, port/RPM/package information, etc.). I have always suspected that percentage of admins who do that is very small, like 1% of people admin'ing open source operating systems. So outside of that, how does one test a patch? I think the "sniff test" is to apply it to one system, reboot that system, and look for obvious, "duh!" errors. For example, you patch the Web server, reboot it (just to make sure that there are no "bombs" in the patch that let it patch fine, but die on reboot), and all of a sudden it stops dishing out Web pages. OK, easy enough. But how does one test for other issues, like one patch disabling a previous patch? Or maybe, say, creating a problem in something that is relatively minor, and you just won't notice it for some time? Say, a scheduled maintenance job is now failing, or maybe an important file had its permissions changed to something that will cause a problem. Another problem along these lines, are the networks with only 1 server. Alternatively, that network may have many servers, but only 1 server per role (1 DNS server, 1 DB server, 1 Web server, etc.). What's the poor admin to do then? On the flip side, all too often, a patch comes out, and 12 hours later, there is an exploit in the wild for the "previously undisclosed vulnerability." I agree, that for systems without direct, unfirewalled Internet access, the concerns are a bit less pressing, especially when you look at the intersection of the problem, software affected, and attack vectors. For example, I turn off SMB on my Web servers in the DMZ, and the firewall won't pass that traffic through, so patches arond SMB are not a huge concern for those servers. On the other hand, they *are* a big concern in my LAN, because I have users who tend to bring things to and from home via USB drives (I've tried to discourage it with little success), and of course, we have a Windows file server in our LAN. Who knows if one of these bugs will migrate from home to office, and then to the file server in a few hours? So yeah, that SMB patch will get applied to my file server ASAP. The risk of an untested patch is much less than the risk of not patching, in my viewpoint, in that particular scneario. Don't get me wrong, I am not against patch testing, not in the slightest. But I would really like to see some good information put out about how to perform patch testing, *especially* for admins in smaller networks who may not have test servers available. J.Ja

Jaqui
Jaqui

After an update to apache broke mod_perl and apache, I started testing every update from a distro. when I started working only from sources, I test the patch effects before deploying the update.

Jack
Jack

Company policy is that systems must be kept up-to-date. Actual practice is that patches get applied only when they fix a known problem. Our systems typically run near the edge of 'unsupported by the vendor.'

apotheon
apotheon

How did you vote in the poll? Why do you use the software update policy you use? What do you recommend for others? Have you ever run afoul of a bad update?

apotheon
apotheon

That's why I included the "something else entirely" choice in the survey. Am I the only person disturbed by the fact that at least seven people out of 135 (5% of the total) said they don't apply any patches at all? That is [b]far[/b] too many people on a Website for people who are supposedly technically competent (IT professionals and their ilk).

apotheon
apotheon

If your options are "patch" and "don't patch", I'd say that "patch" is by far the better option with World of Warcraft. WoW is a particularly scary door into your system for people who take advantage of WoW players, and Blizzard tends to be good about patching vulnerabilities quickly and effectively. Since WoW doesn't interact with other software on the system very much, except under very controlled conditions, and isn't (or at least shouldn't be) used on mission-critical systems that can't stand to be shut down unexpectedly -- so the worst you're likely to see in the case of a bad patch from a reputable source (e.g., Blizzard itself) is crashing or locking up the system. As you mentioned, avoiding plugin sources and the like that might be prone to malware-infected downloads is key to maintaining a secure and stable gaming system.

ITCompGuy
ITCompGuy

I agree with you Justin. I have worked in large and small companies, and I have yet to see a test network that is good enough to perform testing of patches in an environment that can come close to productions. Some people will argue the use of VM's, but VM's are NOT the actual machine and will NOT react to all updates in the SAME manner as the physical machine IN ALL CASES. And how long do you test? I used to build machine back in the day, and we would let them "burn" or run continuously for days to test stability. I administer a WSUS server which is responsible for releasing Microsoft updates to the network. You can configure the WSUS server to automatically apply CRITICAL updates immediately, and to require approval for all others (which is best practices). Never automatically approve ANY update to Servers or other critical systems in your corporation, but rather configure to DOWNLOAD and NOTIFY, which give you the ability to go through EACH update - read the technical information on each from Microsoft - and decide if it is applicable to your situation before approving.

steven@r
steven@r

I really appreciate this discussion, and am glad I am not the only 'clueless puppy' on the block. In my previous job as Programmer/DBA/Network administrator at a small/medium sized law firm with 2 servers, I am not sure how I could have figured out that Microsoft's patch 'C' would break previous patches 'A' and 'B' much less explain any of this to my boss. Regarding Microsoft's descriptions, I agree that most seem to be kind of generic 'we're fixing a security vulnerability', and that doesn't give one a lot of information to go on. The whole subject of 'How to Test a Patch' would be a good subject for a whole other article or even a book. I would buy it.

Jaqui
Jaqui

have one system with every bit of software used by the company on it, server and workstation. apply patches and check basic functionality. run all regular admin scripts / tasks, check for functionality. let it run for a week, check that nothing failed. apply the patches to the production systems unless otherwise indicated.

apotheon
apotheon

"[i]I would really like to see some good information put out about how to perform patch testing, *especially* for admins in smaller networks who may not have test servers available.[/i]" Unfortunately, that's not really all that easy to pull off. As I pointed out in the article, there are times when you have to bite the bullet -- just apply the patch and cross your fingers. One of those times is when you don't have the luxuries of time and test servers. My answer in such circumstances is, as much as possible, to favor software that doesn't tend to have faulty patch issues -- and where rolling back is easier and more reliable when there [b]is[/b] a faulty patch -- as well as where patches don't get pushed out overzealously by a vendor. For an end-user or small-time admin without access to a test environment, there simply isn't any way to test a patch. Choose as wisely as possible, making decisions designed to minimize exposure to bad update effects, maximize the ability to roll back a change, and provide as many recovery options as possible in the event of disaster -- and always seek information as much as it is reasonable to do so before applying updates.

apotheon
apotheon

"[i]In the open source world, I supposed a small number of people inspect the source code (as well as any files related to the install of the patch, such as the makefile's, port/RPM/package information, etc.).[/i]" Actually, software patches tend to get tested rather more thoroughly than that in the open source world. They also generally get applied, and the software gets tested under varying circumstances with the patch applied to check for unwanted or unexpected behavior. Inspecting source code is [b]not[/b] enough. "[i]But how does one test for other issues, like one patch disabling a previous patch? Or maybe, say, creating a problem in something that is relatively minor, and you just won't notice it for some time? Say, a scheduled maintenance job is now failing, or maybe an important file had its permissions changed to something that will cause a problem.[/i]" [url=http://blogs.techrepublic.com.com/security/?p=275]Filesystem integrity auditing[/url] can help with that -- a security measure that, in my experience, is far more commonly used with Unix-like systems than with MS Windows systems. So too can software test suites, which (again, in my experience) are used far more often with open source software than closed source software. "[i]Another problem along these lines, are the networks with only 1 server. Alternatively, that network may have many servers, but only 1 server per role (1 DNS server, 1 DB server, 1 Web server, etc.). What's the poor admin to do then?[/i]" If testing really isn't an option, [b]research[/b]. Also, extend your research time so that others' experiences have time to occur, thus providing better data to research. . . . and in the case of critical security patches, you may have to cut research (just like testing) times short, so it really helps to have a clear and simple way to differentiate meaningfully between important security updates and all other updates. Microsoft's method of specifying whether something is security-related is heuristic and indistinct at best. It is intentionally obfuscated at worst, because of course Microsoft often uses nonstandard definitions for "security" (calling DRM a "security feature", for instance) and is sometimes prone to bundling security-unrelated updates with security updates, or even calling something a security update when such a claim is patently false. At the opposite end of the spectrum, there's [url=http://blogs.techrepublic.com.com/security/?p=477]portaudit[/url] and a very small number of out-of-band VuXML advisory services.

Neon Samurai
Neon Samurai

I've taken to testing all my *nix updates against VMs before production machines also. It was an "update" for phpbb that broke a machine and really cemented the practice for me.

Aakash Shah
Aakash Shah

How do IT admins patch non-MS patches in smaller environments where an expensive patching solution is not affordable? WSUS is free and so is capable of dealing with MS patches, but I haven't been able to find a free or cheap solution to patch third party apps like FireFox, QuickTime, Flash, etc? Any suggestions?

davidt
davidt

I take several steps: 1) Servers are always manually updated after as thorough a research as possible (but testing is not possible - no resources) 2) Certain users I can trust to do the updates themselves (but they are downloaded automatically) 3) Everyone else gets automatic download, automatic install, automatic reboot if required. This has served me well over the years. Some problems, yes, but nothing insurmountable

ITCompGuy
ITCompGuy

I am responsible for administering my companies WSUS (Windows Server Update Service). For most organizations, this is the best solution for a central point of distrubuting updates to end users. It can be a nightmare. The problem with receiving/applying/approving any update is that you have very little knowledge of WHAT WILL ACTUALLY happen when you apply the update. As an administrator I find that I have to spend hour upon hour EVERY week (not just on super Tuesdays), going through information for each and EVERY update that gets applied to the network. Test environments are fine, but how many of us ACTUALLY have a test environment that mimics the production environment? I have worked at large and small corporations and haven't seen a test network yet that was more than just adequate. Some updates are Critical due to some potentially harmful exploit and should be taken as soon as possible (meaning immediately). There is also no proven way of seeing how any update will affect various systems/software/topographies until they are active. I once had a Microsoft Office update that basically shutdown the deployment process for any new machines for weeks. How is that you say? Who knows. I know that it was HECK trying to find the problem. That is because the update had been applied 6 months prior to the revealing of the problem. After many overtime hours, and much time on the phone, Microsoft admitted that they had an "undocumented" problem with an office update that affected WSUS and were working on a patch. They gave a "work-a-round" that we had to keep using until a new "patch" came out months later. There is NO easy solution. Taking updates can be like rolling the dice in Las Vegas or Atlantic City. Sometimes you get lucky, but most of the time you just try to avoid losing your shirt (or your job).

william.r.thomas
william.r.thomas

Tough conversation - I applied the MS patches and they broke x - damn Microsoft. Impossible conversation - I, in my infinite wisdom, decided not to apply the patches because (insert well reasoned decision criteria) and now we have been exploited by x vulnerability let me know how it works out for you....

Neon Samurai
Neon Samurai

Especially on a IT focused website let alone in a discussion hosted under the security section. Now, the trick is to find out who they are and how to get a third party security audit quote on there CIO's desk. ;)

ITCompGuy
ITCompGuy

I am also disturbed by that Apotheon. I look at it as the "head-in-the-sand" syndrome. Things that people do not fully understand or they find too dificult...they avoid and pray that nothing happens. I hope it continues to work for them....

apotheon
apotheon

"[i]The whole subject of 'How to Test a Patch' would be a good subject for a whole other article or even a book. I would buy it.[/i]" To be properly presented, it would pretty much [b]require[/b] its own book. In fact, such books might exist -- you just have to look for books that focus on software security testing, and specifically cover the key subjects of post-patch behavioral changes and interaction with other software. Such books are sure to be presented more for developers than for system and network administrators, however. Trying to cover the subject effectively in an article would be kind of futile. There's a lot more to it than can be provided in an article.

Justin James
Justin James

... provided you can go enough day-to-day tasks on a separate machine. But yeah, I think that's just about the only alternative, even if the separate machine needs to be a VM. J.Ja

Neon Samurai
Neon Samurai

This came up in conversation the other day so I'll risk asking a silly question. How would one roll-back an update on a BSD or Linux based box? Windows offers the pointy-clicky update uninstall feature and driver roll-backs. Would you edit the package black list then uninstall/reinstall to the lesser package version? My current aproach is to test on a VM and if it goes backly, I just reload the clean recovery point. If an update breaks my test box then it's not going on the production box.

Justin James
Justin James

I am just waiting for the final list of desktop PCs to be on the "download and notify list" (as opposed to the "download and install" list) to put this in place. I am already using WSUS, so I can control when the patches actually get pushed out. The combination works. Now, I just need to really nail down how I decide when to release a patch in the first place. :) J.Ja

apotheon
apotheon

I make no guarantees, when I describe the realities of a security situation, that doing the technically correct thing is also going to provide the CYA advantages you might require to keep your job in a top-heavy bureaucracy. There are risks no matter what approach you take -- and you have to evaluate the situation to determine which are the greatest dangers, and what is the best approach to take in your particular situation. If you do it wrong, you might end up looking like the IT department at Sheffield Teaching Hospitals Trust, whose network is [getting hammered by the Conficker worm](http://www.theregister.co.uk/2009/01/20/sheffield_conficker/). Of course, it's worth keeping in mind that the ultimately disastrous decision was made because of the equally disastrous behavior of MS Windows patching, which tended to cause computers used in surgery to reboot at inopportune times.

Justin James
Justin James

Great point. Regardless of what is "right" or "wrong" at a technical level, one reason that applying patches ASAP will likely continue at a widespread level, is because of this. Unless you have a very understanding boss who is willing to accept the risk of an exploit in exchange for doing the testing (and hopefully, signs off on the plan to do just that), you do run this risk. The worst part is, immediate patching is "industry best practice". I've found that deviating fron IBP is career suicide when things go wrong if your boss is not technically strong, because then you have to explain why your "unique" methods are so superior to someone who is feeling the heat from above because the systems failed. J.Ja

tim uk
tim uk

Is it practical to use a VM for performance testing? Often you need to be sure that a patch will not cause any CPU maxing, heavy RAM use, lagging or additional traffic (e.g. if running voip too); are there tools available to test this on a VM?

Jaqui
Jaqui

a vm is a perfect way to go for this. snapshots really useful, since you can do a patch by patch test and roll back by reverting to the snapshot.

garnerl
garnerl

For our Unix systems, we have the snapshot ability. With the data (our Unix servers have little data) on separate partitions or SANS, we can snapshot the OS partition and rollback in a second.

Neon Samurai
Neon Samurai

Ideally I'd like to think there is never a need to roll back a package update but not knowing how to do it ahead of time is just asking for a face full of bad day.

Justin James
Justin James

I know how to *uninstall* software completely... but not how to roll back an update! I don't recall the FreeBSD Handbook mentioning it, either. J.Ja

apotheon
apotheon

Different Unix-like systems employ different means of undoing changes via their default software management systems. On my FreeBSD laptop, for instance, I find this in the ports tree: [i]> cat /usr/ports/ports-mgmt/portdowngrade/pkg-descr Portdowngrade helps to downgrade FreeBSD ports by analyzing the history of commits to the port and presenting the user the list of changes. By selecting one, the port can be set back to a previous version easily. WWW: http://sourceforge.net/projects/portdowngrade/[/i] edit: Notice . . . it even gives hints on how you could roll your own such tool if you wanted to. Nice -- eh?

ITCompGuy
ITCompGuy

Just make sure you only "Auto Approve" Critical and Security related updates. Do not take Service Pack, because you have to plan for how they task your network. Also, Be sure to watch for stuff like IE8, etc. that come through, because they can cause quite a bit of headache, especially when you are not ready to go to the next version due to compatibility issues or development requirements. I used to work for an organizataion that had an extensive Intranet and Extranet that used a bunch of web based applications that DID NOT work with IE7. Microsoft release IE7 as a "critical update" initially, so it created a problem with our users (internal and external) using some webpages.

Editor's Picks