Back in August of last year, you might have read an article here titled, “There’s more to security than counting vulnerabilities.” It made the point that one of the most commonly cited pieces of evidence in debates about how secure some piece of software might be — the number of vulnerabilities reported over a given time period — is in many cases worse than useless as a means of quantifying the security of the software. It’s tempting to look at two software systems, one with three reported vulnerabilities and another with ten over the same time period, and assume the one with three is more secure. In case the previous article didn’t make the point clearly enough, this article tries a different approach to making that point.

A hypothetical example

Let’s put the theory that raw vulnerability counts provide valuable security assessment data to the test with a hypothetical comparison:

  1. FooOS has three vulnerabilities discovered and reported over a two-year period.
  2. BarOS has ten vulnerabilities discovered and reported over a two-year period.

Which OS do you choose? So far, it sounds like you should choose FooOS, if you buy into the notion that more discovered vulnerabilities translates into a less secure system.

Okay, let’s add to the example:

  1. The FooOS developers have patched two out of three vulnerabilities in that time.
  2. The BarOS developers have patched nine out of ten vulnerabilities in that time.

Well . . . maybe we should still go with FooOS, but it’s a little harder to tell. You still have a “winner” on security, though, as long as you continue to imagine that counting vulnerabilities really offers any kind of substantive value in determining which is “more secure”.

A more complete picture

Let’s add some more information about vulnerabilities:

    • Both FooOS patches have required additional patches to fix problems the original patches introduced.
    • The unpatched vulnerability is the first of the three that was discovered, is a flaw in the OS kernel’s scheduler, is almost two years old, and has remained unpatched because the FooOS developers say it’s not actually a vulnerability.
    • The unpatched vulnerability and one of the other two were discovered by malicious security crackers, and became known to the developers only because of exploits in the wild — they were targeted by zero-day exploits.
    • For the two vulnerabilities that were patched, it took close to a month in one case to issue a patch and seven months in the other case.
    • The one that took under a month was reported directly to the developers first, then was reported publicly two weeks later — resulting in the developers smearing the reputation of the guy who discovered the vulnerability, calling him lots of dirty names (like “irresponsible” and “malicious hacker”), because he didn’t keep his knowledge of the vulnerability to himself until after the developers patched it twenty-six days after they became aware of it.
    • In the case of the ultimately patched vulnerability that was not publicly disclosed by the guy who discovered it, the developers kept the vulnerability secret, and millions of users had no way to protect themselves against the vulnerability — despite the fact the developers knew of a simple work-around.
    • None of the BarOS patches required additional patching, because they introduced no new problems.
    • The unpatched vulnerability at the time you perform your assessment was discovered two weeks ago, and actually applies to a third-party piece of software that is typically shipped with the OS, but vulnerability tracking organizations tend to assign such third-party vulnerabilities to the OS itself — even though, in this case, it’s a third-party piece of software that’s easily replaced by a different piece of software that does the same thing.
    • All ten reported vulnerabilities were discovered and reported by users and developers of BarOS who simply wanted the OS to get patched — and, while exploits were developed by malicious security crackers to target two of the vulnerabilities, they only targeted unpatched systems after the patches were issued.
    • Of the patched vulnerabilities, seven were patched within a day of being reported, and two were patched in less than a week.
    • In six cases, the vulnerabilities were reported to the BarOS developers and publicly to the world at the same time, on a public mailing list the developers use for discussing development issues. In the other four cases they were reported privately and not publicly released until after the developers publicly acknowledged the vulnerabilities; in no case did the developers attack anyone for public disclosure of a vulnerability.
    • In one of the two cases where it took several days to patch a vulnerability, the vulnerability was not publicly announced by the person who discovered and reported it — but the developers decided that, since it was a complex problem that might take more than a day to patch and they were really busy that week, they should provide a work-around to the public to protect themselves until the patch was available.

The final assessment

The question remains: Which OS would you choose for security purposes? If you still think that number of vulnerabilities is a good metric for quantifying the security of a system, maybe you’d choose FooOS. Those of us who can see past the tempting simplicity of a single number, however, would probably choose BarOS — all else being equal. In fact, it seems likely from those descriptions that the BarOS developers foster a culture of openness and helpfulness amongst their system’s users, which encourages them to report vulnerabilities they discover so that the developers can fix them, while the FooOS developers foster a culture of hostility between their users and themselves, of deception and secrecy, which might result in people refusing to have anything to do with vulnerability reporting. As a result, it’s possible that FooOS actually had many more vulnerabilities to be discovered than BarOS, but nobody knows that fact because more people are interested in making sure BarOS’s vulnerabilities are discovered and patched.

The lesson to take from this hypothetical example is that counting vulnerability reports is as likely to lead you to the wrong conclusion as to the right conclusion. Find more information before making a decision. Think through the implications of any metric you have available.

Don’t buy the easy interpretation just because it’s easy.