There's more to security than counting vulnerabilities

It's important, when making decisions related to IT security, that you keep in mind not only what options you have and their relation to various measures of security, but also what security metrics are actually useful and accurate. To this end, a security professional must always be careful to avoid measuring security by inaccurate means, regardless of how widely such a metric is used.

Software security is often viewed in terms of reported vulnerabilities. The security industry tends to benefit from such a view because the more people regard reported vulnerabilities as a measure of security, the easier it is for security analysts, software vendors, and auditors to sell their services and products. The idea that vulnerability discovery is an accurate measure of security is generally an easy sell because it's a simple metric and easy to understand. It is a misleading standard, however.

The problems with using vulnerability discovery -- the number and frequency of discovered vulnerabilities -- are many. Among the reasons it serves as a misleading metric of security are the following:

  • Vulnerabilities exist from the moment the software is written -- or, in some cases, are introduced when changes are made to the software, even if some of those changes are intended to fix existing security vulnerabilities. Discovering them only makes the vulnerabilities known but doesn't change the number of them that exist. For a count of vulnerabilities to be an accurate measure of security, it would have to include those that have not yet been discovered, which is a logical impossibility since being able to count them would, by definition, mean they had been discovered.
  • A vulnerability's impact on security, once discovered, is affected by who discovers it and the discoverer's motivation. Many software vendors, especially the largest of them, are most motivated by a desire to appear secure to their customer base. Malicious security crackers can be motivated by money, reputation, or some combination of the two. Security researchers' motivations are much more widely varied. Most open source software contributors, however, are primarily motivated by a desire to ensure that the software they use is as secure as possible.
  • The number of vulnerabilities reported for a given piece of software in no way indicates, by itself, how long it takes for a software distributor to patch a vulnerability. Actual vulnerability exposure is more dependent on how much time passes between discovery and patching of a vulnerability, including the time between release of a patch and when it's applied.
  • Because you can never be certain you know about all discovered vulnerabilities -- vendors may conceal vulnerability discovery, for instance, to maintain the appearance of greater security -- you cannot even be sure you have an accurate count of vulnerabilities. Worse yet, you may suffer greater exposure because without knowing about the vulnerabilities that have been discovered, you cannot protect yourself until a patch is available with workarounds or alternate software choices.

The real value you can gain from knowledge of discovered vulnerabilities is not in judging the essential security of a piece of software. Instead, its value is in providing you with what you need to make informed decisions about how to configure your system to protect yourself, whether by implementing workarounds, altering your software choices, or improving perimeter security and configuration so that various vulnerabilities are effectively irrelevant to the security of the system.

This requires a lot more than mere knowledge of discovered vulnerabilities. Pay attention to how quickly patches are issued for vulnerabilities, how easily they can be applied, the quality of the patches, and how honestly the software distributor handles vulnerability discovery as well. To maintain system security, you have to take all of these factors into account and others as well. Whether this leads to different configurations, different choices of software, or some combination of the two, you cannot make an informed decision without paying attention to all the relevant information.


Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

Justin James
Justin James

The biggest problems that I have with reported vulnerabilities, is that so many of them are found with automated tools. Sure, that fuzzer or tool to try XSS or SQL injection attacks is going to allow you to pick up some "cool boy security hacker points." Hooray. So they get fixed and the reported vulnerability count falls to 0. Meanwhile, the world's easiest exploit can be found by looking at the JavaScript in the HTML source, or maybe looking at the source code for the app, or whatever. My favorite example: last summer, I needed to change a hotel reservation on Hyatt's Web site. It let me use either my confirmation number (which I did not have handy) or my credit card number (which I *did* have handy). I returned to the page later on and noticed that IE was able to autosave my CC number, which it never does for pages that post via SSL/HTTPS. In other words, it was posting my CC number over an unencrypted HTTP stream... BAD! I just happened to notice this (it has since been corrected). No automated tool in the world will catch this, but any cracker will see it in a second. So the automated tool reports 0 errors, a false sense of security. Many of the truly nasty security holes out there will never make it to "reported vulnerabilities list" simply because those lists are constructed nearly entirely with automated script kiddie type tools. Yes, it is important to find and close those holes... but a solid code review by a top coder is still the best way to find the really ugly problems! J.Ja


I hope your voice of reason starts to hit the ears of CIOs and CEOs soon.


Security just like Quality, is built in. Don't forget to mention the physical security of this issue. (or is that for another blog? ;\ ) With every-one looking to a magic technical security solution, it's useually some idiot that just comes through the door that breaks the whole system down. -d (A pet peeve of mine, I know. Sorry :| )


That's for another post. I don't have a specific one in mind right now, but I'm reasonably sure it'll be forthcoming soon enough. I have a lot of other material in the queue before I get to that, I think. Stay tuned.

Editor's Picks