A know-it-all employee of one of my clients told her their perimeter defenses — including firewall — were shamefully inadequate. They needed a NGFW.

I almost said something when my client told me. But for once, I just listened. Besides, what if the employee was right? The not knowing was enough; I started investigating Next Generation FireWalls (NGFW), eventually writing an article about what I learned.

While writing that first article, one thing became evident: NGFWs are complex multi-purpose devices. That makes them difficult to test. You can’t just plug them in and see what happens. Besides, I know this guy.

Need some help

That guy is Rick Moy, CEO and founder of NSS Labs, a research and testing facility for software and digital appliances. NSS Labs caught my attention several years ago. The reason? NSS Labs is completely independent. Rick is adamant about that:

Above all, it means NSS Labs’ mission is to provide IT organizations with the unbiased, test-based truths about products they are spending millions of dollars on. We do not simply ‘certify’ what a product can do; rather we demonstrate what it cannot do, or where its limits are – often to the chagrin of a vendor.

Ultimately, we must be funded by the customers we serve – IT organizations. Unlike other analysts and test labs, we don’t accept sponsorship or funding to produce research or public test reports.

For more on NSS Labs, check out this YouTube video.

Real-world testing

If you watch the video, you may notice something that fascinates me about NSS Labs; Rick and his crew test devices and software in a unique manner. They involve real-world sampling whenever possible. Here’s Rick again:

The key factor in real-world testing is to use live attacks that are currently being used by cyber criminals. NSS Labs’ global threat intelligence network constantly monitors threats in 40 different countries.

This process feeds attacks into our live test harness in which we run products simultaneously in various configurations against the live threats while connected to the Internet. This methodology gives us the most accurate assessment possible of how well protected users are against threats in circulation at any given moment.

To my knowledge, no other testing facility does that. I asked Rick to be sure:

Yes we are unique. First, other testers (labs and corporate IT teams) don’t have the level of visibility or access to global threats. They generally use free or off-the-shelf products with limited capabilities and content.

Our live test harness is completely designed and built in-house. This type of infrastructure is complicated and expensive to both create and operate. There are significant technical and financial reasons why vendor-funded test labs can’t do this in an efficient, automated process.

Now you know why I get in touch with Rick when it comes to firewalls.

NGFW test environment

Testing NGFWs is outside my capabilities, but not NSS Labs’ — as evidenced by the paper Next Generation Firewall Test Methodology. I encourage those seriously considering the purchase of a NGFW to give it a read.

One section of the paper I found particularly interesting is the NSS Labs’ test network.

The NSS Labs test network is a multi-Gigabit infrastructure based on Cisco Catalyst 6500-series switches (fiber and copper Gigabit interfaces).

Traffic generation and receiving equipment — such as the BreakingPoint and Spirent Smartbits transmit ports — are connected to both the “internal” and “external” network.

Attackers are connected to the external network, while vulnerable hosts and Internet-facing servers (Web, FTP, and so on) are connected to the internal network including a DMZ. This enables testing of product performance for multiple scenarios according to product deployment and usage.

NSS Labs appears to be well-prepared for testing. But, I was hoping for more details on their work with NGFWs. Here’s what I found out.
Kassner: You mentioned real-world threats and the network. But, what exactly are you testing in NGFWs?
Moy: In this first big test of NGFWs, we focused initially on baseline functionality, performance, and effectiveness against a controlled set of attacks. These attacks are gathered from the Internet, open source tools like Metasploit and ExploitHub, and other industry partners.

We applied over 1,500 unique exploits against Common Vulnerabilities and Exposures (CVE) that were considered highly critical to enterprise environments using the Common Vulnerability Scoring System (CVSS-7+). There are remote server attacks as well as a large number of client-side attacks against browsers, plug-ins and office applications.

By launching these attacks against target machines on the other side of an NGFW, we monitor whether or not we receive a command shell back to the target machine. If we do, we have gained full control of the endpoint. That’s considered a successful exploit, and a failure of the NGFW.

Without connecting to the Internet we already witnessed a number of issues and limitations that IT buyers must be aware of. Our next phase of testing in Q2 will subject NGFWs to live Internet threats.


Kassner: The paper mentions specific criteria you test, including: Security Effectiveness, Resistance to Evasion, Performance, Stability, and Total Cost of Ownership. Could you briefly define each category? I am particularly interested in “Resistance to Evasion”. What does that mean?
Moy: In addition to the live testing framework, we have a large internal network of vulnerable machines and attack machines we use in security-effectiveness testing. By launching live attacks, we are able to learn which products truly prevent them.

Evasion techniques are methods attackers use to disguise attacks so they can slip past security products that would otherwise detect the original attack. In our tests, we modify the attacks, just like malicious hackers would, and see which products catch both the original and the modified variant.

Our performance testing uses BreakingPoint Systems to push real-world traffic mixes across devices at speeds exceeding 80 Gbps. But throughput is only part of the equation. Latency impact, connections per second, and maximum concurrent connections that can be achieved are crucial factors for data center and perimeter devices.

Stability testing is one of the most grueling aspects of our tests. We run a battery of protocol fuzzing and mutation tests over multiple days. In our NGFW test, half of the products initially failed in some way.

Finally, our Total Cost of Ownership analysis takes a number of factors into consideration. Value after all is more than just cost. We also look at:

  • How much security is obtained per Megabit/sec.
  • Can a product scale without impacting staffing requirements.
  • Does the management console and approach require additional operators to monitor and process alerts.

Kassner: You tested several models of NGFWs; which ones were they and how do interested parties obtain the test results?
Moy: We invited all firewall vendors to submit products at no cost. We received devices from Barracuda, Check Point, Fortinet, Juniper, Palo Alto Networks, SonicWALL, and Stonesoft.

The research, test results, RFP tools, and analyst services are available on a subscription basis. Interested parties can learn more by visiting our website.

Kassner: Rick, you have considerable experience with first-generation firewalls — are NGFWs a big improvement? Should companies seriously consider replacing their existing systems?
Moy: Next Generation is a marketing term, and it does not necessarily mean “better” for every company and use case. Companies need to consider their needs seriously, and whether or not a NGFW is required or even suitable.
For example, many NGFWs have performance and latency characteristics that make them poor choices for certain types of applications and data needs; e.g., market trading data, or real-time low-latency applications.

Interestingly, many of the products demonstrated quality issues typical of first-generation products: stability, performance, and management difficulties. In many ways, NGFWs are completely new products. You cannot simply glue a firewall to an IPS and expect it to go smoothly. Add application ID plus control and it becomes even more challenging. Vendors have had to rethink their architectures for these products.

Final thoughts

Note to self, remember to thank a certain client’s employee. His remark — and my ego — better positioned me to advise the client. She can then decide which, if any, NGFW best fits the business, based on NSS Lab’s testing.

Rick and I thought it best to point out that I have no affiliation with NSS Labs other than shamelessly bugging him with all my questions.