When you talk about external penetration (pen) testing with a group of security managers, the discussion can get a little heated.  There seems to be a lot of passion about whether pen tests are a waste of time or a necessary tool for security controls management.  I believe the effectiveness, or not, of pen testing is related to approach and expectation setting.

Adherents to the pen-tests-are-useless position assert that it’s the penetration testers who benefit most (financially) from this futile exercise.  Next in line are security managers who either need proof for the auditors that they are actually looking for vulnerabilities and those who use the myriad vulnerabilities discovered to justify additional budget.  This group of naysayers believes that the best approach is to simply deploy policies, standards, guidelines, and infrastructure as part of a well designed security program.

On the opposite side are the security managers who believe that testing is necessary.  After all, how can you identify weaknesses in your perimeter if you don’t test for them?  Further, they contend that it is important to show management and the auditors that you’re looking for and correcting all potential attack paths to your information assets.  This includes a complete scan of all ports and services across all public-facing devices.

I fall between these two positions.  Yes, I believe that pen testing with no clear agenda is a waste of time.  I also believe that the lack of focused testing is negligent. 

A pen test should focus on an organization’s public IP addresses, scanning for the top known vulnerabilities (e.g. the SANS Top 20).  The purpose of the test should be to validate that the steps taken to secure the perimeter are “good enough”.  I never expect to eliminate all vulnerabilities.  However, I do strive to reduce risks from these vulnerabilities to a point where business impact from a successful exploit is at an acceptable level. 

I also don’t care about the raw data.  We engage a managed security services provider to perform an automated scan as well as aggregation and correlation of collected data.  We’re then presented with a portal view of what the data mean.  This allows us to compare quarterly scan results to a baseline and to expected results to verify that our controls are working as expected.

The cost of an automated penetration test isn’t very much, but it provides valuable validation of controls.  Although we’re required by our auditors to perform a pen test, we don’t hold up the results as proof that our network is absolutely secure.  There will always be some level of risk.  Rather, we portray the portal information as just one more input into our continuous security improvement efforts.