There's more to security than counting vulnerabilities

It's important, when making decisions related to IT security, that you keep in mind not only what options you have and their relation to various measures of security, but also what security metrics are actually useful and accurate. To this end, a security professional must always be careful to avoid measuring security by inaccurate means, regardless of how widely such a metric is used.

Software security is often viewed in terms of reported vulnerabilities. The security industry tends to benefit from such a view because the more people regard reported vulnerabilities as a measure of security, the easier it is for security analysts, software vendors, and auditors to sell their services and products. The idea that vulnerability discovery is an accurate measure of security is generally an easy sell because it's a simple metric and easy to understand. It is a misleading standard, however.

The problems with using vulnerability discovery -- the number and frequency of discovered vulnerabilities -- are many. Among the reasons it serves as a misleading metric of security are the following:

  • Vulnerabilities exist from the moment the software is written -- or, in some cases, are introduced when changes are made to the software, even if some of those changes are intended to fix existing security vulnerabilities. Discovering them only makes the vulnerabilities known but doesn't change the number of them that exist. For a count of vulnerabilities to be an accurate measure of security, it would have to include those that have not yet been discovered, which is a logical impossibility since being able to count them would, by definition, mean they had been discovered.
  • A vulnerability's impact on security, once discovered, is affected by who discovers it and the discoverer's motivation. Many software vendors, especially the largest of them, are most motivated by a desire to appear secure to their customer base. Malicious security crackers can be motivated by money, reputation, or some combination of the two. Security researchers' motivations are much more widely varied. Most open source software contributors, however, are primarily motivated by a desire to ensure that the software they use is as secure as possible.
  • The number of vulnerabilities reported for a given piece of software in no way indicates, by itself, how long it takes for a software distributor to patch a vulnerability. Actual vulnerability exposure is more dependent on how much time passes between discovery and patching of a vulnerability, including the time between release of a patch and when it's applied.
  • Because you can never be certain you know about all discovered vulnerabilities -- vendors may conceal vulnerability discovery, for instance, to maintain the appearance of greater security -- you cannot even be sure you have an accurate count of vulnerabilities. Worse yet, you may suffer greater exposure because without knowing about the vulnerabilities that have been discovered, you cannot protect yourself until a patch is available with workarounds or alternate software choices.

The real value you can gain from knowledge of discovered vulnerabilities is not in judging the essential security of a piece of software. Instead, its value is in providing you with what you need to make informed decisions about how to configure your system to protect yourself, whether by implementing workarounds, altering your software choices, or improving perimeter security and configuration so that various vulnerabilities are effectively irrelevant to the security of the system.

This requires a lot more than mere knowledge of discovered vulnerabilities. Pay attention to how quickly patches are issued for vulnerabilities, how easily they can be applied, the quality of the patches, and how honestly the software distributor handles vulnerability discovery as well. To maintain system security, you have to take all of these factors into account and others as well. Whether this leads to different configurations, different choices of software, or some combination of the two, you cannot make an informed decision without paying attention to all the relevant information.