Serious admins don't ignore security patches

With a virus threat, there are two kinds of Admins: those who are alert and apply patches and those who think it better to do nothing and instead cast blame.

By scOrp

ScOrp is TechRepublic's first opinion writer. His ideas and suggestions are his own and do not represent those of TechRepublic or CNET Networks.

Between the twin-threat events of this month—the power blackout in the North American East and the latest worm (news of which seemed to hit everywhere but The Jerry Springer Show), call it "Blaster" or "LovSan" as you will—it was an interesting time to be in charge of electronic information flow (mileage may vary according to one's personal geography).

The moving finger points…
As far as the blackout went, Republicans blamed Democrats for balking at the President's energy plan, Democrats blamed Republicans for including "unconscionable" energy policies in that plan, and Canadians blamed all Americans without regard to political bent. THAT debate will be ongoing for some time to come, unlike the clarity associated with placing responsibility for the LovSan worm. Clearly, it was ALL Microsoft's fault, that one...

...until you look at the facts. Looking at the facts usually changes everything, except peoples' minds.

…and having pointed, wags
Somewhere around the end of last June, a team of Polish software experimenters calling themselves the "Last Stage of Delirium Research Group" found and demonstrated a really big hole in the RPC implementation of DCOM services. They got in touch with Microsoft and coordinated a joint news release, along with a nice new patch for the hole.

In a week, only 40 million patches had been downloaded. Also in a week, real live code to exploit the RPC hole was posted on the Web. The countdown began, and in a week, there were reports of unexplained shutdowns and other wobbly behavior. In another week, the worm was found, isolated, and dissected by the F-Secure Group in Finland. Over the next two to three days, things really got interesting in Internet forums. Once again, the IT biz had made the six o'clock news and people were experiencing machine shutdowns at home and at work. It was getting personal.

The bare-bone facts are these:
  • A major security hole was found and reported to Microsoft.
  • In short order, the news was spread, without referencing the specifics of how to exploit it, and a patch was published.
  • A small percentage of systems were patched.
  • Someone figured out how to exploit it and published the code.
  • Someone else (unknown at this time) used that same code as the base to insert a worm as a carrier for a time-bomb DoS attack through the RPC hole.
  • A side effect of the RPC exploit was to cause system shutdowns (pure sloppiness; done right, it should have left no trace of its presence).
  • Soon, and in time for patches to be emplaced before the DoS attack was to execute, a major media blitz occurred, giving everyone notice.
  • Patched or not, everybody blamed Microsoft for carelessness.
  • The DoS attack never happened, partly through an inaccuracy in the code; the URL was faulty, and Microsoft shut down the target URL anyway.
  • Everyone continues to blame Microsoft.

It is sometimes not very inspiring to defend Microsoft, but it has to be done; they did very much the right thing once notified by LSD. There were two kinds of Admins (and home users): Those who were alert and applied the patch (or allowed Windows Update to do it automatically) and those who thought it better to do nothing and instead cast blame. The first thing that occurs to me is that the second group was very lucky that this worm carried nothing more damaging than a simple zombie. The second is that there could still easily be another worm more artfully written to avoid the signature shutdowns that alerted people that there was a problem.

There is a balance to be found with patching, of course. "Best Practices" calls for downloading and testing, and ensuring that the patch can be backed out of. Ideally, you have a test network that simulates the production environment, with all operating systems and applications present, and run it through paces with the patch applied, before deploying the patch to the production net. Second best is to wait—how long is up for debate. Some patches have been found to have their own set of problems that may or may not apply in individual cases. Waiting a week and watching for field reports of the patch is likely a sound practice. Waiting a month or more is probably not.

The point is that the serious Admin does not simply ignore such warnings as the one Microsoft issued in this case.

Nonpatching whiners
Yet a look at Internet forums and at the comments after posted articles indicates that this is exactly what happened in many cases. Servers and workstations went down and production suffered, all for the lack of applying this patch. (Again, aren't they lucky that the payload was so relatively benign?) The whining going on is rather appalling to behold. Much of the whining gets slapped down on the spot by the Admins (and a few home users too!) who stayed alert and patched like they were supposed to.

In the aftermath of the Dot Bomb and the reorganization of the IT career scene, there are four more or less clear groups of people in two major categories: those with an IT job and those without. (Another breakdown that probably covers the majority is "those who wish they had an IT job" and "those who wish they didn't".) The first group (the whiners) are irritating to everyone else, but in different ways.

To the (apparent) minority—the Admins who took the time and did the job—the nonpatching whiners give the field an undeserved bad name. It's a relatively high-paying job that makes demands on the holders of it; their feeling is "shape up or get out." This group may be larger than you may think; most of them are too busy reading their newsletters and bulletins and planning for the next strike to happen, and don't bother commenting upon issues which to them are merely obvious and part of the job.

There are the wannabes, those at the Help Desk with an MCSE and those working at a gas station with an MCSE who never even scored a Help Desk job. Many of these people saw fortune and glory in IT and stumbled through their cert studies (or worse!). Others simply can't imagine really having any career that is NOT in IT; they caught the bug. They see the whiners and are enraged... because they just KNOW they wouldn't have been caught. There is likely some real talent lurking in these ranks. There are likely a lot of people who should stay at the gas station as well. Whether or not the wannabes will ever get their chance is up in the air.

Then there are the Dispossessed—those who have a year or more of experience in the IT field and were laid off when the gutting of the ranks occurred. Either unemployed or underemployed, they see the whiners as worthless slackers and obstacles to The World As It Should Be—one attribute of which is (naturally!) that they get their old jobs back. Again, there is real talent among these folks; some of them lost their jobs through pure office politics or local economic misfortunes. Others should have been fired. As a side note, gone are the days when doing one thing well meant job security.

The Whiners themselves? Whether it's "If only the boss would let us run Linux!" or "Microsoft should do it right the first time!" or "We never had this problem with Groupwise!" or "We're understaffed and overworked...." or even "We run Linux so NAHH NAHH!"—the answer is simple: Quit your job.

Here's my point.

You had all the time in the world to download and test this patch, and deploy it out. You do not have the excuse that you were not told; this bulletin came out a month ago with specific instructions and an explanation of the severity of the issue. You can't even claim that someone brought in an infected laptop and hooked it up to the network after you patched. Look closely at this KB article—ever seen that one?—and you'll even find that Microsoft even made available a scanning tool to run daily, and catch those invading laptops. The audacity you show in exposing your slack attitude (you think your boss never reads those comments?) is matched only by your incompetence.

Quit your job. Stop making life harder for those who do theirs (let alone your company and its users), and make way for someone who wants to do it right. Hopefully you can make French fries.

Editor's Picks

Free Newsletters, In your Inbox