Dealing with emergency patches to fix critical vulnerabilities is about as much fun as paying taxes. It's yet another fire drill which interrupts more meaningful work, but the process also ensures said work can resume via secured systems and applications.
Because there's no way to avoid emergency patches, here are seven tips to help make patch distribution easier:
1. Research the vulnerability and the fix
Not all vulnerabilities are equal. Despite the fanfare that may erupt, some exploits may pose less of a threat than assumed, or may not apply at all in your environment. If you're running HP servers and a problem with Dell firmware is announced, obviously you don't need to bat an eye. If the exploit requires internal access from a compromised system and you're using firewalls to segregate traffic between networks, you have less to worry about (although you may not be out of the woods entirely since some systems may still be impacted, and as a rule critical patches should eventually be applied across the board).
Similarly, if the vulnerability has to do with a web browser — say, Internet Explorer — and what might happen if it accesses a certain external site, if all your production systems are blocked from accessing the internet you don't have to worry about those being affected.
Read up on what the fix does, what side effects it may cause, and whether a rollback is possible. It's rare but I have seen some infrequent instances of patches causing issues on systems (both Windows and Linux) which then provoked a removal. I've also seen manual efforts required to tear out unwanted patches, such as removing registry keys and system files.
2. Test the patch in advance
Testing the patch before rolling it out is one of the most important things you must do. The systems which you use for testing should closely mirror what's in your live production environment. It does you no good to install a patch on a bare-bones Windows system nobody uses then say "Yep, everything looks good." The test system should have the same accounts, applications, settings and configuration as the machines your business depends on.
Also, don't just reboot and log in and pronounce the patch clean. Check to see if applications run, services have started, users can log in, if there are any odd errors in the logs, whether performance is impacted, etc. Come up with a workable set of strategies for testing systems based on their role or function. If you patch an Exchange server, you need to do more than confirm whether people can send mail; is anything missing, do calendar invites work, and do archiving rules still apply?
3. Plan and prepare the rollout
Don't blindly release a patch to everything at once; make a plan to roll out the patch to affected systems based upon their priority and the potential severity of impact. Obviously production systems should come first, as they may invoke the greatest amount of damage if exploited or compromised. Or, in the example from tip #1, systems with internet access (read: user workstations) should be patched if there are external threats lurking.
If you have redundant systems, such as clustered database servers, patch them separately to ensure success. If you patch them at the same time and they break, your cluster is now dead in the water, defeating the purpose of redundancy.
Make sure to notify IT support staff of any possible issues and their corresponding fixes as well, so they will be aware and can assist with resolving these problems.
There may be some systems you simply can't patch at this time. Perhaps you're in a peak processing period whereby even brief downtime would pose too much hardship, a known issue is preventing the system from successfully rebooting, or the patch would break something else. Depending on your security team's requirements, you may be able to enact what's known as a compensating control (also known as an alternative control) which is essentially a workaround to meet security requirements too onerous or unfeasible to implement. For instance, remove internet access from a system (servers rarely need this) to ensure it won't connect to malicious websites which might then exploit it.
4. Conduct an orderly patch release
Distribute the patches per the plan you devised. Make sure to test your systems post-reboot for the elements or criteria they depend on. This should be conducted in stages with realistic and reliable delivery dates. If one section of the plan gets held up, the entire rollout gets impacted so it's more important to be methodical than rush through to achieve 100% compliance using unrealistic deadlines.
5. Monitor for success
Any good patch management solution will provide the ability to run real-time reports to see whether systems have received the patch and are thus compliant. Check these immediately after patch release and on an ongoing basis to determine a count of which systems still need the patch. Share and discuss with the appropriate personnel as needed.
6. Address the stragglers
Gathering a list of non-compliant systems means you have to act upon it as well. There are many reasons why a system might not receive a patch as deployed. Perhaps the system was down or being repaired, there may have been a communication or authentication problem, the hard drive is out of space (good system monitoring should prevent that), or any number of other issues. Figure out what went wrong and why, then correct it and deploy the patch to close the loop.
7. Hold a postmortem and adjust the patching process as needed
Conduct a review with appropriate staff to see how the process worked. What went well? What could have gone better? Were there any gaffes which were preventable? Was too much or too little information sent out? Are there underlying issues with the process, the communication or the compliance levels which need adjustment?
Implement the recommendations or determinations from the prior step to make the process flow better next time and reduce delays, pain points, miscommunication or other impeding factors.
Because, as every user of technology knows, there WILL be a next time.Also see:
- Microsoft patches Shadow Broker exploits: Make sure you apply these updates
- How misaligned incentives give hackers an advantage over IT security pros
- Think that domain you're visiting is legitimate? Think again: Homograph attacks fake secure domains
- 6 common enterprise cybersecurity threats and how to avoid them
Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.