Security

Vulnerability counting revisited: a hypothetical example

Vulnerability counting is, in many cases, worse than useless as a means of quantifying the security of the software. I've made this point before, but this article tries a different approach to making it: demonstration by hypothetical example.

Back in August of last year, you might have read an article here titled, "There's more to security than counting vulnerabilities." It made the point that one of the most commonly cited pieces of evidence in debates about how secure some piece of software might be -- the number of vulnerabilities reported over a given time period -- is in many cases worse than useless as a means of quantifying the security of the software. It's tempting to look at two software systems, one with three reported vulnerabilities and another with ten over the same time period, and assume the one with three is more secure. In case the previous article didn't make the point clearly enough, this article tries a different approach to making that point.

A hypothetical example

Let's put the theory that raw vulnerability counts provide valuable security assessment data to the test with a hypothetical comparison:

  1. FooOS has three vulnerabilities discovered and reported over a two-year period.
  2. BarOS has ten vulnerabilities discovered and reported over a two-year period.

Which OS do you choose? So far, it sounds like you should choose FooOS, if you buy into the notion that more discovered vulnerabilities translates into a less secure system.

Okay, let's add to the example:

  1. The FooOS developers have patched two out of three vulnerabilities in that time.
  2. The BarOS developers have patched nine out of ten vulnerabilities in that time.

Well . . . maybe we should still go with FooOS, but it's a little harder to tell. You still have a "winner" on security, though, as long as you continue to imagine that counting vulnerabilities really offers any kind of substantive value in determining which is "more secure".

A more complete picture

Let's add some more information about vulnerabilities:

    • Both FooOS patches have required additional patches to fix problems the original patches introduced.
    • The unpatched vulnerability is the first of the three that was discovered, is a flaw in the OS kernel's scheduler, is almost two years old, and has remained unpatched because the FooOS developers say it's not actually a vulnerability.
    • The unpatched vulnerability and one of the other two were discovered by malicious security crackers, and became known to the developers only because of exploits in the wild -- they were targeted by zero-day exploits.
    • For the two vulnerabilities that were patched, it took close to a month in one case to issue a patch and seven months in the other case.
    • The one that took under a month was reported directly to the developers first, then was reported publicly two weeks later -- resulting in the developers smearing the reputation of the guy who discovered the vulnerability, calling him lots of dirty names (like "irresponsible" and "malicious hacker"), because he didn't keep his knowledge of the vulnerability to himself until after the developers patched it twenty-six days after they became aware of it.
    • In the case of the ultimately patched vulnerability that was not publicly disclosed by the guy who discovered it, the developers kept the vulnerability secret, and millions of users had no way to protect themselves against the vulnerability -- despite the fact the developers knew of a simple work-around.
    • None of the BarOS patches required additional patching, because they introduced no new problems.
    • The unpatched vulnerability at the time you perform your assessment was discovered two weeks ago, and actually applies to a third-party piece of software that is typically shipped with the OS, but vulnerability tracking organizations tend to assign such third-party vulnerabilities to the OS itself -- even though, in this case, it's a third-party piece of software that's easily replaced by a different piece of software that does the same thing.
    • All ten reported vulnerabilities were discovered and reported by users and developers of BarOS who simply wanted the OS to get patched -- and, while exploits were developed by malicious security crackers to target two of the vulnerabilities, they only targeted unpatched systems after the patches were issued.
    • Of the patched vulnerabilities, seven were patched within a day of being reported, and two were patched in less than a week.
    • In six cases, the vulnerabilities were reported to the BarOS developers and publicly to the world at the same time, on a public mailing list the developers use for discussing development issues. In the other four cases they were reported privately and not publicly released until after the developers publicly acknowledged the vulnerabilities; in no case did the developers attack anyone for public disclosure of a vulnerability.
    • In one of the two cases where it took several days to patch a vulnerability, the vulnerability was not publicly announced by the person who discovered and reported it -- but the developers decided that, since it was a complex problem that might take more than a day to patch and they were really busy that week, they should provide a work-around to the public to protect themselves until the patch was available.

The final assessment

The question remains: Which OS would you choose for security purposes? If you still think that number of vulnerabilities is a good metric for quantifying the security of a system, maybe you'd choose FooOS. Those of us who can see past the tempting simplicity of a single number, however, would probably choose BarOS -- all else being equal. In fact, it seems likely from those descriptions that the BarOS developers foster a culture of openness and helpfulness amongst their system's users, which encourages them to report vulnerabilities they discover so that the developers can fix them, while the FooOS developers foster a culture of hostility between their users and themselves, of deception and secrecy, which might result in people refusing to have anything to do with vulnerability reporting. As a result, it's possible that FooOS actually had many more vulnerabilities to be discovered than BarOS, but nobody knows that fact because more people are interested in making sure BarOS's vulnerabilities are discovered and patched.

The lesson to take from this hypothetical example is that counting vulnerability reports is as likely to lead you to the wrong conclusion as to the right conclusion. Find more information before making a decision. Think through the implications of any metric you have available.

Don't buy the easy interpretation just because it's easy.

About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

29 comments
saghaulor
saghaulor

You wouldn't be referring to Windows and Linux, or IE and Firefox would you?

apotheon
apotheon

I was referring to OSes, so it definitely wasn't IE and Firefox in this case. It wasn't [b]specifically[/b] related to MS Windows and Linux distributions. In fact, there are cases where Linux distributions have acted more like FooOS in the past (though very, very rarely). I drew inspiration for FooOS from several sources (one of which was MS Windows, of course), and for BarOS from several sources (including various Linux distributions and BSD Unix systems, including FreeBSD). So . . . no, I wasn't referring to those examples specifically. If the shoe fits, though, feel free to wear it.

saghaulor
saghaulor

Yeah, apparently I missed the obvious clues that you were only referring to OSes. I brought it up because I remember reading something about IE and Firefox, and which one was more secure. Some site listed one as more secure based on the amount of vulnerabilities it had in comparison to the other. I remember thinking that the amount is mostly irrelevant depending on all the rest of the circumstances surrounding said browser vulnerabilities. Anyways, thanks for the article, it was very insightful.

apotheon
apotheon

"[i]Yeah, apparently I missed the obvious clues that you were only referring to OSes.[/i]" No biggie. Mistakes happen -- even to me. Ssshhh. Don't tell anyone I said that. "[i]I brought it up because I remember reading something about IE and Firefox, and which one was more secure. Some site listed one as more secure based on the amount of vulnerabilities it had in comparison to the other. I remember thinking that the amount is mostly irrelevant depending on all the rest of the circumstances surrounding said browser vulnerabilities.[/i]" I've seen such comparisons before. In fact, back when flame wars about the comparative security of Firefox and IE were more common here at TR, I saw claims that IE was more secure because of an imbalance in reported vulnerabilities between the two during a three month period. I tried to make the point several times that counting vulnerabilities isn't the be-all and end-all of security evaluation, but I'm not sure there were many people paying attention. Biases made them blind, I guess. "[i]Anyways, thanks for the article, it was very insightful.[/i]" You're welcome -- and I appreciate the positive review.

JCitizen
JCitizen

I've always taken the holistic approach. But this article made me think about it more. Thanks apotheon! FooBar! Ha! =)

jmgarvin
jmgarvin

Great read and a must read for anyone in IT, but especially for the decision makers to understand what they are buying when they buy FooOS vs BarOS (which I kept reading as BeOS).

seanferd
seanferd

The Blue-eyed OS. But I've been messing around with a Haiku VM for a while, so I suppose BeOS was somewhere "in the back of my mind" waiting to be called forward. Then again, FooOS seems like Fool'sOS, as pronounced by Mr. T.

apotheon
apotheon

Even though I wrote the article, I've glanced at "BarOS" a couple times and seen "BeOS" instead. There's just something about the way "BarOS" looks, I guess, that lends itself to that impression. Thanks for the compliments -- and thanks to everyone else who offers such praise for the article, too.

shardeth-15902278
shardeth-15902278

foo actually has a 4th vulnerability, that they pawn off as being a problem with a third party app. Each company blames the other. Ultimately the third party fixes the issue within their product. But the original vulnerable mechanism in foo still exists, waiting for some other bad 3rd party app to make it an issue again.

apotheon
apotheon

I should have thought of that one as well.

Jaqui
Jaqui

bang on the target. It is more important to have vulnerabilities FIXED than to have low numbers reported. the third os option you didn't mention farOS shall we call it had two reported vulnerabilities in that same time period. the first one: they patched it within a week. they discovered it themselves while performing a security audit. they reported it publicly immediately. the second one was reported to them by a user, and their response matched that of the barOS developers to the user. they were able to get a patch out in a day for it. would they be a better option than barOS? or fooOS? just to add the third option for OS and style of work, since as we know, there is at least one os option that matches farOS. edit: to remove yyy typo in style

apotheon
apotheon

In short, you really need to think about what a statistic means before you use it to inform your decision-making. In many cases, the intuitive interpretation is the wrong interpretation.

pgit
pgit

In defense of Microsss... er, FooOS, the count most people pay attention to is populated with a lot of vectors that exploit the same (single) vulnerability, 'duplicates' as it were. There may be a dozen iterations of a virus but one exploit underneath them all. I use BarOS, btw.

jdclyde
jdclyde

that FooOs doesn't fix the underlying vulnerability, allowing multiple avenues of exploiting that issue? And the ONLY fair comparison is if you run both OS's without any third party packages to protect them. The tables do not look nearly as even then.

apotheon
apotheon

"[i]I had read Chad's article Truth about Viruses before, as far as I'm concerned that's a minor issue, I'm more turned off by the corporate model...[/i]" Well . . . it's basically the corporate model that [b]creates[/b] the problem to which I referred in [i]The truth about viruses[/i]. edit: I just wrote a little bit about how the corporate model interferes with ethical behavior, in an essay titled [url=http://sob.apotheon.org/?p=444][i]Corporate Responsibility[/i][/url].

pgit
pgit

Love the avatar, I suspect my cat does that behind my back. As I said I use Linux, I see it as far more secure than I would argue in a forum like this, don't want to be responsible for the inevitable flame war. I had read Chad's article Truth about Viruses before, as far as I'm concerned that's a minor issue, I'm more turned off by the corporate model...

apotheon
apotheon

As I pointed out in [url=http://blogs.techrepublic.com.com/security/?p=286][i]The truth about viruses[/i][/url], much of MS Windows' problems with viruses is the fact that Microsoft refuses to address many virus-exploitable vulnerabilities. It's also worth noting that most of the vulnerabilities we see reported for MS Windows aren't those virus-exploitable vulnerabilities. They're vulnerabilities that Microsoft actually addresses (most of the time, eventually), while most virus-exploitable vulnerabilities never even get reported as vulnerabilities.

shardeth-15902278
shardeth-15902278

At least that I am aware of is patch deployment, specifically in companies which must meet strict guidelines for FDA/ISO/GMP (IQ/OQ/PQ process). More patches can mean substantial cost to run through the process more frequently. I am curious as to how others manage this - if anyone cares to share.

apotheon
apotheon

One tool I'm glad to have in my toolbox for managing patch deployment is [url=http://blogs.techrepublic.com.com/security/?p=477][b]FreeBSD's portaudit[/b][/url]. I think I'll write another article about some more tools and techniques I use at some point in the near-ish future.

jdclyde
jdclyde

Ease of compromise. Severity of exploit. Time from provider finding out to they time they warn the users, offering a work-around until a fix can be found. Turnaround for a fix. Does fix remove all future similar exploits of option?

koen.bossaert
koen.bossaert

Indeed, I had the same comments, you should take the risk rating into account and if lessons are being learnt. It would also be better to relate the number of vulnerabilities to the number of lines of code, if you realy still want to compare numbers.

apotheon
apotheon

The reported criticality of most vulnerability reports is pretty suspect, in general. For instance, in many cases a vulnerability is reported as being of a very low threat rating because it can only be exploited once you already have unauthorized access to a machine -- such as many privilege escalation vulnerabilities that are nominally only exploitable locally. Of course, if you gain access to a low-privilege account on the machine remotely, you may then be able to make use of the privilege escalation vulnerability to turn a largely harmless security compromise into complete ownership of the machine. Vulnerabilities cannot really be reasonably examined for the level of danger they represent without considering them within the context of the rest of the system's security context. Trust the criticality ratings of vulnerability reports at your own risk -- especially when those reports are made available by a vendor with a vested interest in making its software seem more secure than it is.

seanferd
seanferd

Now copy/paste article to ZDNet for forum comments. ;)

apotheon
apotheon

Nobody has given me republishing access at any other C|NET properties, so I don't really have access to republishing it at ZDNet. That would certainly get some comments, though, if I did. Mostly uninformed comments, I'm sure.

seanferd
seanferd

Neon Samurai pretty much nailed it. I was wondering just how it would go down in the NBMer/ABMer warzone. (Look out! May contain actual facts!)

Neon Samurai
Neon Samurai

Then it would be interesting too see how the discussion evolves there versus how it has here. CNet was my daily stop for more mainstream type tech coverage; the forums where lively but carried a lower level of knowledge due to not focusing on tech professional traffic.

Editor's Picks