Last month (July 2009), NSS released test results comparing how each of the major Web browsers dealt with socially-engineered malware. I was going to write about it then. But, every time I came close to posting, new information came to light. It’s finally time to sort this out.

What is socially-engineered malware

My friends Google and Wikipedia were of no help. Finally, about a third of the way through the report, NSS defined socially-engineered malware as:

“A web page link that directly leads to a ‘download’ that delivers a malicious payload whose content type would lead to execution.”

I get it. Socially-engineered malware is referring to malicious or compromised Web sites containing dropper programs. That’s a good test; dropper programs are currently one of the most successful method of infecting computers.

What’s being tested

Modern-day browsers automatically check the reputation of Web sites before allowing content to be downloaded. The report explains how:

“The foundation is an in-the-cloud reputation-based system which scours the Internet for malicious websites and categorizes content accordingly; either by adding it to a black or white list, or assigning a score (depending on the vendor’s approach). This may be performed manually, automatically, or some combination thereof.

The second functional component resides within the web browser and requests reputation information from the in-the-cloud systems about specific URLs and then enforces warning and blocking functions.”

To put it simply, NSS is checking the quality of each Web browser’s malicious-URL data base, how long it takes the database to be updated with new information, and how the Web browser reacts when a match is found.

Test results

NSS screened a total of 12, 000 malicious URLs, finally deciding on 608 URLs that met their requirements. During the test, NSS introduced a certain number of the chosen malicious URLs every day, recording each Web browser’s ability to block the threat. The first graph shows the percentage of malicious URLs each browser successfully detected and blocked (courtesy of NSS):

NSS also recorded whether the Web-browser’s database contained information about each threat. If information about a specific threat was missing, NSS kept track of how long it took before the database was updated. Those results are shown in the following graph (courtesy of NSS):

Meaning what

The graphs bode well for Internet Explorer 8 when it comes to blocking socially-engineering malware URLs. NSS ran similar tests looking at how each Web browser blocked phishing URLs and Internet Explorer 8 was on top again.

Many security analysts are concerned that Microsoft paid for the tests. Evidently, Microsoft’s on-line security-engineering team hired NSS to run the benchmark tests. In fairness to Microsoft, Rick Moy president of NSS mentioned to Ars Technica that:

“This stuff is expensive to do right, and we need to monetize it somehow. We invited Google, Mozilla, Apple, and Opera to participate, but they didn’t even bother to respond, except for Opera, which stated they don’t really focus on malware.”

Final thoughts

Are the tests valid? Consider the following:

  • NSS is not saying much about the malicious-URL list.
  • NSS is not telling why it left out certain exploit sites.
  • Microsoft paid for the tests.

Update (19 Aug 2009)

I presented Rick Moy (president of NSS ) some of the questions you the members were asking:

1. Where did NSS obtain the list of 12,000 malicious websites?

“We obtained Sites from our own research, crawling, spam traps, etc. As well as from other parties not involved in the test – eg Mailshell, Sunbelt. No vendor had access to test URLs prior to the test.“

2. What were the exact criteria for including a website in the test??

“Socially engineered malware as defined in the report. This is an attack on the user, not an exploit on the software. That is a future test.”

Here is the definition: A web page link that directly leads to a ‘download’ that delivers a malicious payload whose content type would lead to execution. I was confused by the definition, until Mr. Moy explained. The user has to consciously click on a link to start the exploit process.

3. A website can use “social engineering” (without Javascript required) to persuade visitors to select a hyperlink that will cause the browser to download a malware installation package and execute it. Were any such websites in the test suite??

“Yes. “

In explanation: NSS defines socially-engineered malware as the process of enticing the user to click on a link.

4. Were the malicious URLs linked to fake malicious Web sites or subverted official Web sites?

“We saw both.“

5. How much input did MS have in the process?

“This test was part of a recurring test program we have been running for about 9 months. All the browser vendors and some AV vendors had a chance to review the test methodology and make comments. Final decisions made by NSS exclusively. We encourage everyone to review the methodology and decide for themselves if this reflects a reasonable real-world test. We will review and consider all comments. Note that to date we have received no substantive critique on this test methodology.”

Kudos

I would like to thank the members for providing the questions, with a special nod to Ocie3. I also would like to thank Mr. Moy for taking the time to answer them.