Antivirus applications rely on malware signatures, antivirus programs are not proactive, antivirus software is ineffective. Sound familiar? I’ve succumbed to the mantra myself, writing — “How antivirus software works: Is it worth it?” and “Traditional antivirus software is useless against military malware.”

If that’s true, why do the same reports slaying antivirus applications end with the CYA — never ever leave your digital home without AV software. Case in point, my final thoughts on the first article linked above:

Being one of those “rather be safe than sorry” types, I will continue to suggest using an antivirus program.

So what gives?

I decided to ignore the conjecture, the innuendo, even my own, and ask an expert. Preferably, one who tests antivirus applications day in and day out, maybe even the chairman of some testing organization.

It just so happens I bumped into such a person; his name is Simon Edwards. He is the Technical Director of Dennis Technology Labs, and — current chairman of AMTSO, the AntiMalware Testing Standards Organization..

To be honest, AMTSO wasn’t on my radar. Which is odd — the member list is a veritable “who’s who” in the antivirus industry. I found it interesting to see testing organizations and vendors working together.

I checked out a few of the AMTSO organizational documents — pretty in-depth stuff. For example, I found a document that spent 19 pages debating the pros and cons of creating malware specifically for testing:

One of the most hotly-debated issues in the antimalware industry today is the question as to whether it is ever right to create a new piece of malware for the purpose of testing antimalware software.

There it is. We have the controversy, and we have an expert that knows AV software inside and out. Let’s see what he has to say.

Kassner: Welcome Simon. One of the first things I’d like to clear up is whether antivirus software and antimalware are the same thing or not, and which term does the industry prefer?
Edwards: That’s an interesting question. All so-called “antivirus” software does far more than detect and remove viruses. Companies only use this phrase because consumers are familiar with it.

To be honest, AV software often handles more than malware — blocking phishing attacks for instance. It’s probably more realistic to describe it as “internet-security software,” but that’s so non-specific I can understand why firms stick with antivirus.

Kassner: Good enough for me, we’ll stick with antivirus as well. Simon, you wear two different, yet related hats — that of Technical Director of Dennis Technology Labs and Chairman of AMTSO. Could you tell us about each, starting with Technical Director?
Edwards: Dennis Technology Labs specializes in internet-threat testing, having the required methodology and expertise in this challenging area. As Technical Director, my main task involves ensuring tests are conducted properly. Another focus of mine is developing new tests to address the latest challenges. I personally have been testing antivirus software for well over a decade.

Testing antivirus software is hard to do well. So, unfortunately there is a lot of poor testing going on. The result is confusion for consumers, and frustration from antivirus vendors. To try to address this situation, all of the best-known vendors and testers formed the non-profit organization AMTSO.

AMTSO’s mission is to promote the best testing possible. A good test is one that is unbiased towards vendors, the methodology is transparent, and the results are both meaningful and scientific. All testers should want to achieve these goals. I was voted in as Chairman last year.

Kassner: IT professionals and security pundits are saying antivirus software is a lost cause. It’s based on a reactionary model, thus destined to never meet expectations. How would you respond to those concerns?
Edwards: A lot of criticism toward antivirus solutions is based on an out-of-date view of how these products work. For example, some people assume “antivirus software” is just a simple file scanner.

In the old days, malicious files were analyzed, after which a “definition” would be added to the antivirus database. This was distributed to PCs running the scanner, which would then be able to detect the new file as being malicious.

This is far from reality today. Modern antivirus products include behavioral components, file and website reputation systems, and a variety of other layered defenses. Some include exploit-code blocking, which is particularly effective.

Are these all reactive? To an extent, yes, it is impossible to predict which websites criminals will compromise next. Also, malicious files change fast. However, many attacks use the same toolkits, so products that support exploit detection and blocking should be able to stop these attacks, even after the criminals tweak their settings.

Kassner: As someone who is intimately familiar with antivirus technology what concerns you the most about the current malware versus antivirus situation?
Edwards: While general attacks that affect everyone are a concern, targeted attacks are the real challenge facing providers of antivirus products.

Take these two scenarios as examples:

  • An attacker compromises a website and causes it to infect any visitor who loads the webpage. The malware that infects their PCs will steal bank details and other valuable personal information.
  • An attacker sends emails to senior managers who work in a particular area of industry. These emails contain links to an infected website that is of no or little interest to the general public. It may not even be indexed by a search engine.

In the first example, people will be infected. A few may report their experiences to a security firm, either by sending an email or by allowing their antivirus software to report back automatically.

The firm will then examine the threat and take steps to ensure that all of its customers are protected. It is common for antivirus companies to share this type of information with competitors, spreading the umbrella of protection across as many internet users as possible.

In the first example, the threat has a relatively short lifespan, because the threat affects many people, therefore quickly comes under the scrutiny of security professionals.

In contrast, the second example is a subtle threat. Only a few people are exposed to the targeted attack, and the downloaded files may never be noticed as being problematic.

Kassner: For those of us who aren’t experts, yet interested in the process, could you explain what happens when a developer sends Dennis Technology Labs an antivirus program to be tested?
Edwards: We take a forensic approach when testing antivirus software. We don’t trust what the products claim. For example, if we visit an infected website and the product claims to have blocked it, we still check low-level details of the PC we are using to ensure that nothing has slipped by.

Sometimes a product will miss a threat to begin with. But, once the bad code is running on the system, the antivirus may kick in and remove it. We examine the extent of the impact the threat has before judging the product’s effectiveness.

For example, did the threat run? If so, what changes did it make to the system? Did it steal any information? Are any important system files damaged or otherwise altered? Did it deactivate the antivirus software?

Kassner: Dennis Technology Labs belongs to AMTSO. Removing your AMTSO hat for now, what influence does AMTSO have on how you test antivirus products?
Edwards: In pre-AMTSO days, when we tested antivirus software for Dennis Publishing’s computer magazines there was no incentive to share any detailed information with the vendors. AMTSO changed our view on that.

By being transparent, and describing everything that our tests uncover, we not only establish that we are doing what we say we are, but we help antivirus vendors improve their products. This in turn helps consumers.

Kassner: I’d like you to put your AMTSO hat back on. When we look at any one of the myriad antivirus test reports, what should we look for?
Edwards: There is a list of guidelines that AMTSO has published to answer that question. For me one of the things that jump out at me from a poor test is when the conclusions are at odds with the test data. For example, a test takes a selection of exploits and demonstrates that an antivirus product fails to detect them. It then concludes that antivirus is useless.

It may be fair to conclude the product has limited exploit detection, or that its offline file scanner is sub-standard, but to write off the whole product (and sometimes these tests are used to write off the whole industry) is disingenuous.

It’s fair to say there is no such thing as the perfect test but if you want a list of things with which to judge a test, you could use the AMTSO Fundamental Principles of Testing.

Kassner: I’m betting you are asked this all the time. But, I promised to include the question. What should we look for when shopping for antivirus products?
Edwards: Consistency. It is all very well choosing a product that comes out on top of one test, but look at multiple tests from the same and different organizations. If two or three products consistently perform well, and in tests conducted by different testers using different methodologies; then it’s likely a strong contender.

The same goes for products to avoid. If it consistently under-performs, even if it’s free, avoid it.

Final thoughts

Don’t worry; I’m not going to use my CYA statement this time. We all know that AV software is not the end-all solution. It’s a Band-Aid. But — and help me out here — what are we going to do in the mean time?

Thank you, Simon for shedding light on a complex problem, and providing insight in how to make sure our current solution is working correctly.