While recently working on a network with roughly 10,000 hosts, I discovered two machines that had gone unpatched for 11 months. This might not seem like a big deal -- .02 percent of machines that remained unpatched doesn't sound too bad.
But it only takes one unprotected machine to infect or disable an entire network. That means one unpatched workstation is a problem.
If you still question whether this really is an issue, here are some of the potential repercussions:
- Downtime and loss of productivity due to reinstallation
- Questionable data integrity due to a successful exploit
- Negative public relations due to systems unavailable for your customers
- Legal problems if your patch management process has to go under a judicial microscope
While patch deployment tools have become standardized fare on many midsize to large networks, there still remains a problem that many fail to notice or address: Some machines will always go unpatched.
No tool is going to solve this problem for you. Patching is a physical process -- not a tool. Some organizations treat patch deployment as a task. It's not a task; it's a full-time job.
Solving this problem takes several proactive steps. Here are five steps to managing the patch deployment process.
Start by taking an inventory of your organization's network and the software deployed on it. Without this, you won't know which patches to deploy. In addition, prioritize machines by creating a risk profile based on their necessity to the organization.
Once you have an inventory and prioritization of your assets, you're ready to start monitoring for patches. This is usually as simple as signing up for e-mail notification on the product vendor's Web site.
But don't rely on your vendor to always give you timely notification of problems. It may not notify you until it's developed a solution, and that could be too late. That's why you need to also monitor security Web sites for news of zero-day exploits and emerging problems.
After you've set up a monitoring process, it's time to start deploying patches. There are many patch deployment solutions available, so pick one that fits your enterprise's needs and budget.
All solutions deliver patches. However, what you need most from your patch management system is a report on what the solution patched and which machines it did not patch.
Now it's time to remediate. Using the report of which machines failed, track them down, and start monitoring them for activity.
Some users habitually turn off workstations at the end of the day regardless of how many e-mail notices they receive to leave them on. Monitor your network for activity from those machines.
Sometimes drastic action is necessary to protect your organization's network. You can always disable the machine's account and send a note to the help desk to route the trouble ticket through the patch management section.
If this doesn't work, then disable the switch port that the workstation connects to. Either solution will produce a response from the user and allow you to solve the short-term problem of deploying the patches. Then you can start working on the long-term problem of why this workstation is failing to receive patches.
While automating patch management is possible, it's not always the best solution. It's important to inform management of the seriousness of such a situation and ask for the resources to do your job. The goal is zero failures -- it only takes one to bring down your entire network.
Mike Mullins has served as an assistant network administrator and a network security administrator for the U.S. Secret Service and the Defense Information Systems Agency. He is currently the director of operations for the Southern Theater Network Operations and Security Center.
Worried about security issues? Who isn't? Automatically sign up for our free Security Solutions newsletter, delivered each Friday, and get hands-on advice for locking down your systems.