Security

12 security suites tested and 12 security suites fail

Chad Perrin warns users and admins against the lure of one-size-fits-all security suites and offers advice on building an effective defense of your network.

In September of 2007, I wrote "The truth about viruses" and pointed out how the ubiquitous danger of viruses exists largely because of negligence. When the vulnerabilities that common viruses exploit never get fixed, and those viruses are only guarded against in a case-by-case manner using signatures-based and heuristic detection systems, new viruses that will bypass detection and still affect your computer can be created by the hundreds and thousands with minimal effort. In short, much of the reason for the ubiquitous threat of viruses is a tendency of software vendors to ignore virus-exploitable vulnerabilities, and expect antivirus vendors to pick up the slack.This is not a problem particular to viruses. In fact, a good antivirus application can protect you from viruses reasonably well most of the time. Those of us who deal with security issues professionally, or even regularly as a hobby, are understandably leery of the idea of being "reasonably well protected" from something "most of the time." Still, it's obvious that antivirus software is not a complete failure as a Band-Aid over a sucking chest wound.

The same problem exists outside of virus-exploitable vulnerabilities, however, and is not nearly so well addressed. As Gregg Keizer reports in "Top security suites fail exploit tests," integrated security suites for desktop computers fare much worse across the range of threats against which they're expected to protect you.

I've prepared a couple of simple bar graphs to give you an idea how much they protect you against virus threats and active attacks. I cut the number of compared vendors down to 10 in each case, because that's the number of vendors that overlapped in the two shootouts. In both graphs, I will use the color green to show threat coverage that exceeds 50%, yellow to show threat coverage that exceeds 25% up to 50%, and red to show threat coverage no higher than 25%. In both graphs, they're ranked from best performing vendor to worst performing.

The first example is from the June 2008 Virus.gr antivirus software shootout, and in each case where a single vendor had more than one product in the shootout, I counted only the best-performing product:

Antivirus Performance by Vendor

The second example is from Secunia's [PDF] October 2008 Internet Security Suite test:

Security Suite Performance by Vendor

An antivirus application is expected to do well at protecting against viruses. While I wouldn't consider anything lower than, say, 98% coverage to qualify as doing sufficiently "well" to satisfy me personally, at least nobody came in under the 50% wire.

An integrated Internet Security Suite is expected to protect one against active threats; it should include effective firewall, rootkit detection, active vulnerability defense, and at least some rudimentary kind of real-time intrusion detection. Sadly, one might have noticed I didn't get to use my virtual yellow highlighter at all in that security suite graph. Everything came in below 25%. Even though the best was significantly better than second place for the vulnerability prooofs of concept that made up the testing gauntlet, it was nowhere near good enough to even bother.

The problem here is multifarious. A few key points include the following:

  • Notice that there isn't much relationship between who were the best performers on one graph and who were the best on the other. A lesson to take from this fact is the idea that maybe companies that are good at one thing aren't necessarily good at another. The best performer on the AV graph was tied for third worst on the security suite graph; the best performer on the security suite graph was fourth worst on the AV graph. The fact some vendor seems to do well at AV in no way suggests you should entrust that vendor's software with all your security software needs.
  • As pointed out in Secunia's test results document, vulnerability defense coverage is abysmal in every case -- more so in some cases than in others. When corporate software vendors do not quickly and effectively patch vulnerabilities (which is almost always the case) and users do not test and apply security patches in a timely manner (which is usually the case), vendors for security software should definitely look into picking up some of the slack. Can you imagine the marketing benefits if you were the only vendor to achieve more than 50% coverage in Secunia's test -- especially with second place, an order of magnitude better than third, scoring less than half as well as you?
  • Security software vendors shouldn't even have a vulnerability defense market to target. Software vendors should be patching vulnerabilities more quickly than security software vendors can develop active defenses for them, and software update systems should be designed to make it at least as easy to find and apply patches without breaking functionality as to fail to update. This is a huge challenge for closed source software vendors, of course, but that doesn't mean it's not a challenge that needs to be met.

I can offer three simple pieces of advice, if you want effective defense, one to deal with each of the above problems:

  • Build your defenses a piece at a time, selecting the best security options for each part of the whole. Don't trust a single vendor to get everything right, because, frankly, it probably won't.
  • Track vulnerabilities yourself. Choose software that offers good mechanisms for doing so, and protect yourself to the best of your ability.
  • Make your software selections at least in part based on good vulnerability response time (and don't fall into the trap of simply counting discovered vulnerabilities).

About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

52 comments
tranieri2
tranieri2

That both of the graphs are labeled the same, but have different results?

gladhatter
gladhatter

I am a bit confused why you did not put Zone Alarm to the test as well. I own Bit Defender and do not even use it anymore and now I see that it would be usless basically anyway. I would really like to know how ZA scores here all around. I am now considering reinstalling Hacker eliminator again so I can at least monitor all new processes. This is just horrible

Jaqui
Jaqui

"insecurity" suites fail? The vendors don't want your system to be secure, if it is, you have no need of their product / service.

apotheon
apotheon

Do you entrust your security to a security suite?

apotheon
apotheon

The reason ZoneAlarm wasn't included in the charts is explained in the article.

Neon Samurai
Neon Samurai

The best thing to happen to the AV and malware protection rackets since virus writers and script kiddies.

boxfiddler
boxfiddler

I love you, santeewelding. Yuppers.

OnTheRopes
OnTheRopes

I've tried them in the past but was unsatisfied. One of their failings, in my opinion, is that they require too much interaction. I'm thinking about the average home user; I believe that they're just as likely to allow the wrong thing as not. It's interesting to me that Secunia's test box wasn't a fully patched box and that they had some vulnerable programs installed. Too bad they didn't say what programs were included and what patches were missing.

The Scummy One
The Scummy One

this shows that users NEED to be more in control of their own security. They need to be better informed and actually give a damned about their online behaviour.

w2ktechman
w2ktechman

just how well doubling up does? For AV alone I use AVG pro (paid) and Avast free. Both ranked high in the AV category (and neither were on your list). I am sure that one will ID or correct a problem that another does not. But what about full suites? Would they just be more resource intensive, or would 2 good performing ones act like a very, very good suite? Oh, and I was shocked at Sophos's performance. I use that on Linux, and although these were tested in Win, it still shows it as very lacking.

rkuhn040172
rkuhn040172

Sure, Microsoft products seem to suffer from the vast majority of system flaws, however, one has to recognize that anything created by a human can be hacked by another human...we aren't perfect. Therefore, while most of what Chad said are true, I think he unfortunately takes too much of a negative tone towards blaming the industry for not trying hard enough. When any OS consists of millions and millions of lines of code, any OS will have vulnerabilities that are yet to be discovered.

apotheon
apotheon

Unfortunately, there are no one-size-fits-all suggestions. My recommendations would be mailing lists -- specifically, lists for the software packages you're using (e.g., Apache, MySQL, Debian), the security basics list, a LUG list or two, and probably a couple of others as you find things that seem relevant. You might also want to look into a couple of reputable "hacks" books (such as from O'Reilly's Hacks series, particularly Network Security Hacks), the O'Reilly Linux Server Security, Linux Security Cookbook, and Network Security Assessment. Don't take those as authoritative, for the most part: use them as jumping-off points for further inquiry. Of course, you don't have to look in all of those sources in *particular*. That's just the sort of thing I'd recommend, because they're reasonably good sources for the kind of information I'd use as a jumping-off point myself if looking into details on how to set up a Webserver for a purpose outside my usual needs. Good luck with it.

santeewelding
santeewelding

Kitty Stutter A teacher is explaining biology to her 4th grade students. 'Human Beings are the only animals that stutter,' she says. A little girl raises her hand. 'I had a kitty-cat who stuttered.' The teacher, knowing how precious some of these stories could become, asked the girl to describe the incident. 'Well', she began, 'I was in the back yard with my kitty and the Rottweiler that lives next door got a running start and before we knew it, he jumped over the fence into our yard!' 'That must've been scary,' said the teacher. 'It sure was,' said the little girl. 'My kitty raised his back, went 'Sssss, Sssss, Sssss' and before he could say 'Shit,' the Rottweiler ate him! The teacher wet her pants laughing. (submitted by Jim, whose incoming mail I went to check)

santeewelding
santeewelding

I can envision em and en quads, and the old, grizzled operator rolling his one good eye as he makes the slug bearing my "multifarious" slide down and clunk in the tray. I envision the "m" the way a sign painter tutored me long ago: "Seems everybody learned how to hand print the 'm' and the 'w' the same, wrong way. The middle peak at the bottom in one and the top in the other are best the same as either side." Inspecting illuminated manuscripts laying open at Oxford comes to mind. "Nefarious" comes to mind, but the Latinate "multifarious" rolls off the tongue in that stentorian way. I would go on with "whole" and its relation to "multi", but I would probably lose you and you would level the common accusation that I am being incomprehensible.

shardeth-15902278
shardeth-15902278

The "Let us take care of it for you." Approach that most software vendors, and many IT shops use, just isn't going to work. Users need to be empowered, informed, enabled, and involved.

shardeth-15902278
shardeth-15902278

I received an email recently, containing a zip file purporting to be an account report from a vendor. Inside the zip file was a .exe cleverly disguised as a word doc file. (file.doc.exe, with a .doc icon embedded. To most windows users, it would just appear as file.doc, thanks to MS default hide known extensions.) It was obviuosly (to me) a viral package, but I decided to run my AV's (clam, AVG) against it, to see what it was. Nothing reported by either. I uploaded the virus to a clearinghouse site (checks against the databases of about 20 different AV packages. None of the well known packages registered it. 4 unknown (to me) systems, registered it as "suspicious", or "unknown malware". I assume they did some form of heuristic analysis. (I really should'a wrote their names down). Anyway my point is. You can't depend on (any) AV to protect you. In some respects one could argue that you are better off NOT running AV, so that you won't have a false sense of security, and become complacent, trusting it to protect you. edited to fix a bunch of spelling mistakes

Greenknight_z
Greenknight_z

AVG Technologies was formerly named Grisoft, they changed it in February 08 (AVG stands for AntiVirus Grisoft, I guess). Avast is a major omission, though. If you run both those AVs, I hope you have the real-time protection shut down in one of them - two AVs running at the same time can interfere with each other and cripple your protection. I wouldn't even risk having two such apps installed; I use Dr. Web CureIt as a backup scanner, it has no real-time shield.

apotheon
apotheon

"[i]would 2 good performing ones act like a very, very good suite?[/i]" Before you could find out, you'd have to track down two suites that actually performed worth a damn. At the moment, it looks like there isn't even one of them in the world. . . . and, regardless of whether it worked well at all, it would definitely drag most computers to a glacial crawl as it consumed a metric crapload of system resources. Doubling up is pretty much the only sane approach to antivirus, by the way -- but you have to be careful about what AV software you use so that the two programs don't get into a fight over resources, et cetera.

Neon Samurai
Neon Samurai

I won't condem MS just for being the most profitable but I still think they could do a great deal to improve there products. They are trapped by there own success and history but there seems to be more importance on maximizing profits rather than improving product quality. Much of that millions of lines of code is for legacy reasons that should be reconsidered. A better balance between profits and expenses would leave more budget for quality control. This is still the company that can afford to employ some of the brightest minds in the industry yet it remains crippled by it's bureaucracy and culture. Other OS consisting of millions of lines of code manage to provide the higher degree of quality. Forgetting the popular Linux family of OS, Unix systems alone prove that MS could do better in design decision making.

w2ktechman
w2ktechman

that is basically what I got out of it.

shardeth-15902278
shardeth-15902278

1. I don't see that he targeted MS specifically, there are plenty of (non-OS) vendors in scope as well here. 2. I don't see that he targeted anyone for undiscovered vulnerabilities. He was specifically speaking of known vulnerabilities, which aren't being fixed in a reasonable time period (or ever). The fact is (as he seems to be saying [but I have misinterpreted before]) much of today's security software is a an attempt to squash the bugs crawling through the holes, and 1) They are doing a amazingly crappy job of it. 2) It would seem any i'jut would fairly quickly conclude that plugging the holes is a much more sensible approach.

apotheon
apotheon

"[i]Sure, Microsoft products seem to suffer from the vast majority of system flaws, however, one has to recognize that anything created by a human can be hacked by another human...we aren't perfect.[/i]" We can't ever achieve 100% success -- so why bother trying for 12%? "[i]Therefore, while most of what Chad said are true, I think he unfortunately takes too much of a negative tone towards blaming the industry for not trying hard enough.[/i]" I'll try to adopt a more positive attitude toward industry leaders not trying hard enough. "[i]When any OS consists of millions and millions of lines of code, any OS will have vulnerabilities that are yet to be discovered.[/i]" [url=http://blogs.techrepublic.com.com/security/?p=268][b]Indeed.[/b][/url]

Neon Samurai
Neon Samurai

I was planning a book store run to see what is applicable that I don't already own. In general, I find a minimum of three separate books on any topic is required to have a good starting towards understanding. I think mailing lists for specific major applications uses will be a must also. My first concern is keeping the three machines healthy so that justifies dedicated reading time to keep up with the subscribed email chatter. (I'll be reactivating my securityfocus subscriptions too now that I have a way to manage the inflow of mail.) Thanks Apotheon. I've thought of these tips in the past but it always provides clarity to be reminded of them by someone else.

boxfiddler
boxfiddler

perfect for a Friday Yuk. Copied and pasted, if you don't, I will. Providing someone doesn't beat both of us to it.

OldER Mycroft
OldER Mycroft

If I'm an expert a$$ripper, I've got to be given technical instructions. :)

boxfiddler
boxfiddler

You rip new @**holes with ease, I've noted. ;)

OldER Mycroft
OldER Mycroft

Oblige. It's not often that I get the opportunity to let rip on TR with something that I truly am an [b]expert[/b] on. [i]I did edit my prose regarding your M's and W's.[/i] Thanks for the opening. :)

santeewelding
santeewelding

Worked, too. No confusion on your part, except that I did specify "hand" printed "m" and "w". What you went on about with respect to that was your own reverie, which I fully intended to provoke.

OldER Mycroft
OldER Mycroft

Or is that just your way of hoping to confuse all TR peers who were not Techies in the ancient Print world? Of course you realise that the [i]"grizzled operator rolling his one good eye as he makes the slug bearing my "multifarious" slide down and clunk in the tray"[/i] is most likely six or seven sheets to the wind mentally. His physical condition is not far behind due to the excessively high levels of [(Pb+2,Sn+2)6 Fe+2 Sn+4 2Sb+3 2S-214] in his bloodstream. As for the 'M' and the 'W', if it were being printed, you'd only have that problem when you allowed sloppy returns in the Caseroom. Actually, come to think of it, the leading on the furniture and the augmentation of the quoins should prevent that mistake from happening. Otherwise the composite type wouldn't fit the Chase! But, hither - I dither, letting my mind race back to a time when men were men and women were housewives (or so the old Linotype Operators used to proudly tell me). I remember one old bloke who used the hot metal pot, to heat up his Scotch Pie that he ate at lunchtime - he reckoned it heightened the flavour!! :^0 [[u]For those of you that Santee was trying to confuse:[/u] [b](Pb+2,Sn+2)6 Fe+2 Sn+4 2Sb+3 2S-214 = Lead Tin Iron Antimony Sulphide, the chemical compound of liquid hot metal, forced into Brass Die's, used in the production of Linotype slugs. Each 'Slug' was an entire line of type, hence the name Linotype. Individual characters, assembled by (some reckon 'human') Compositors, was called Monotype.[/b] [i]The Linotype company went on to introduce the first electronic photo-typesetting system called the LinoComp, then the first PC-based computerised typesetting system called the LinoTron. They were also the inventors and patent holder for the Postscript system of electronic typefaces, later licenced to Adobe.[/i] Consider that, the next time you feel like complaining that your keyboard doesn't work! Happy Days..... ;) [u]Further Reading:[/u] http://www.linotype.com/49-14026/19731989.html

boxfiddler
boxfiddler

If only you were the slightest bit moreso... You would lose me.

The Scummy One
The Scummy One

gotten 2 virus on any of my personal systems, over the last 5 years (that I am aware of). However browsing is often a problem as I dont allow javascript without my consent, nor flash at all. I'll allow flash in FF on Linux though, for the times I really want to get something done and cannot without it.

Neon Samurai
Neon Samurai

You where suspicious and considered that it was not an expected attachment. There is no compensating for intelligence by any number of blinky lights. Out of curiosity, what was the malware or where you not able to identify it ever? Also, what was the clearing house site if it is open or is it a subscribed service you maintain?

marketingtutor.
marketingtutor.

As a developer, I really have to agree. AV Software is useless in targeted attacks. It's great in broad;y distributed generic viruses. But when you're the target of corporate theft, ID theft, etc, there are many ways to bypass the alarms and security through the tainting of legitimate programs. Sure, stateful packet inspection and MD5 application checksums help, but still, if the threat comes from a trusted source, nothing will ever work. Do I run a firewall/security suite, yeah, Comodo's rubbish. But for any real threat assessment, I have a VMware sandbox snapshot that I run the potential threat app in before running it on my target machine. Record all of it's actions, and make sure it's safe first. Not a perfect solution, but it works for me. I also surf from a virtual machine that I restore to a snapshot at every startup. Haven't had a virus affliction in many years.

apotheon
apotheon

"[i]In some respects one could argue that you are better off NOT running AV, so that you won't have a false sense of security, and become complacent, trusting it to protect you.[/i]" I don't run AV on any of my own computers at the moment -- but then, I'm running systems where the antivirus strategy is to [url=http://blogs.techrepublic.com.com/security/?p=286][b]patch the virus-exploitable vulnerabilities[/b][/url], rather than to just trust AV software vendors will cover their backs. In essence, AV software is basically just redundant and obsolete on the system I use.

NotSoChiGuy
NotSoChiGuy

While I wouldn't agree that people shouldn't run AV in order to increase awareness; I do agree that far too many users have too great a reliance on the AV software or other network protections (firewall, content filters, etc). To a large extent, the technologies are always chasing the malware/crackers; so you're going to be looking at some window of possible exploitation, however small and brief it may be. You're also relying on other people, technically, to protect you (via updates to software, releasing new DATs, etc). Just like driving; no matter how careful you drive, it just takes one numnut on the road not paying attention to cause you to get into a crash.

Neon Samurai
Neon Samurai

As Apoth mentions, it is designed to be a standalone scanner not an active scanner. Now, ClamAV (the unix build) can be run through many different plugins. Stick it on your IPS/IDS or gateway with the plugin to monitor traffic flow, or mail server plugin, or samba plugin, or apache plugin and you may have some value in considering it. It may not be the right solution for everyone but it is worth considering if your infrastructure as unix boxes protecting all the win32/64. The only variable I see still is the signature files. I have to look into that for myself in detail soon. It's on my work task list.

apotheon
apotheon

Clam(AV/Win) is not designed as an "enterprise" desktop antivirus solution. If you use it that way (without some kind of external management system), you're doing it wrong. There are other purposes for which Clam(AV/Win) is ideal, however. Calling it "dead on arrival" just because you aren't running a system for which it would be ideal is pretty damned narrow-minded of you.

rkuhn040172
rkuhn040172

Too many people here talk about personal protection. But in a corporate environment, Clam lacks the management features necessary to properly monitor a network.

Neon Samurai
Neon Samurai

It had Norton originally but ran out of signature file updates so someone put McAfee on it to get the later signatures detectable. Since config, uninstall and updates where all locked out by a lost admin password, Norton was left in place and the next reboot meant calling me in to fix it. I think both where versions previous to one being smart enough to disable the other. My bet today would be that the second install would disable the first as part of the setup wizard. The real betting money is in which Malware illness smart enough to disable both will be contracted first when left booted and directly connected to the ISP. :D Now the machine uses a different AV active scanner.

w2ktechman
w2ktechman

have never played well together. Talk about a bad match for doubling up! I am surprised they didnt try disabling each other the first boot after install of the second suite. Hmm, might be fun to watch -- which will disable the other first?

Neon Samurai
Neon Samurai

The AV chapter used Clam as the training example and went into detail on how one builds there own signature file to account for new mutations not yet included in the official signature files. I've heard good and bad about it though I'm glad to hear it has a good reputation (or did) again here. Either way, it's free, it doesn't eat resources and it has plugins to run it against most standard server deamons so there is no issue including it in system builds. I think I'll have to do some reading on it in the next few weeks and see what the latest reports say for it though. Offhand, and barely related, any good sites to read on what general functions should be included in a server build outside of the deamons for what the server is specialized to do? I've a webserver and two database servers to build. The basic http and sql deamons are an easy pick. The variables are in what security and monitoring software to include. Deb is the choice of distribution since it has a focus on stability and security (the OpenSSH issue not withstanding). It also has bastille to confirm that I haven't missed anything. SELinux support if we go that far. There is also the list of external audit apps I'm going to hit it with during the testing stage. I'm absorbing all I can as fast as I can but I'm always on the lookout for recommended books, websites or other information sources on hardening and pentesting.

apotheon
apotheon

"[i]I'm also not sure how much updating Clam gets[/i]" I recall a few years ago the news being good on that score, but I haven't checked back since then, and things could very easily have changed. I just don't know. My work these days has nothing to do with selecting desktop AV solutions.

apotheon
apotheon

On another shootout a couple years ago, ClamAV was in the top 10 out of about 50 participating AV solutions. In a survey of thirty or so AV applications, ClamAV was in the top five for virus signature development response times. The relative position really does vary significantly between these comparisons of AV solutions. What doesn't tend to vary is the range of coverage percentages -- between 50% and 98% percent for the vast majority.

Neon Samurai
Neon Samurai

I'll keep Clam as a curtesy to Windows platforms but that's a good reminder to keep me from considering it alone rather than as a backup check.

Neon Samurai
Neon Samurai

I'd like to think the signature files are kept very up to date but I've never had the chance to test that in real life. I know one of it's strengths is being able to write your own signatures for it to include with the downloaded ones but virus signature identification is not an every day admin skill (sadly). On the up side, the Unix install can be run as a active scanner deamon or at least triggerd by other apps as needed. procmail, samba, apache.. I usually watch for the plugins now. It may not be the latest signature dat files I'm getting during my regular check but at least it's looking for the stuff it does recognize.

Jaqui
Jaqui

They infest a system with known malware and run the scan. clam isn't as effective at catching even older known malware as the others. I asked them about clam two years ago and they explained the testing criteria in their response to me. :)

apotheon
apotheon

Since ClamAV isn't as security suite, it wasn't in the suite shootout. It didn't do all that hot in the AV shootout, but then, I don't know what the test criteria were for that one; I didn't look into the testing methodology. For all I know, they might have been doing the equivalent of testing tire traction in a rainstorm, with Clam brand tires being the equivalent of racing slicks. In other words, for instance, they might have been testing realtime scanning capability as part of their criteria -- and ClamAV/ClamWin is a scheduled scan application only.

Neon Samurai
Neon Samurai

The last time I saw two active scanners on the same machine it went very badly. Bootup started and things seemed to go well. Eventually McAfee or Norton loaded and things stopped going well. As far as I could tell: McAfee would see a file opened, capture that and start the active scanning process. McAfee opening a file would notify Norton's active scanner so norton would in turn grab that same file and start the active scanning process. The file being opened by Norton would activate McAfee's active scan process and around, and around.. I had to do some gymnastics to get into the system without the two AV programs loading. Once one was removed, I could reboot and gain enough control to remove the other. Two "on access" AV on the same machine wasn't my doing but it was my fixing that made the box usable again. Since then, I've only ever considered one active AV on a system though schedualled scan AV along side are not such a bad thing. With portable ClamAV and the website hosted scanners, you can get multiple checks done. (Now I have to go check the graphs and see if Clam was even included and if so, how badly did it do compared to the other failures)