Cybercriminals have a significant advantage over the good guys. “Our adversaries have an incredible environment for testing attacks: the internet,” writes Terry Benzel in this National Science Foundation press release. “They can sit and analyze our vulnerabilities for as long as they want, probe and poke and run experiments until they find the right way in.”

What’s more, the bad guys do not care what they break or whom they hurt.

Benzel, deputy director for the Internet and Networked Systems division at the Information Sciences Institute of the University of Southern California, and other like-minded individuals at DETER (cyber DEfense Technology Experimental Research) have for the past 12 years leveled the digital playing field by providing cybersecurity researchers a similar proving grounds without having to worry about wrecking the internet.

SEE: My interview with Terry Benzel for TechRepublic in 2011


To make that possible, Benzel and her team designed, built, and now operate DETERLab, a digital test bed where business, government, and university researchers work with DETER in-house experts to analyze cybersecurity solutions.

In this YouTube video, Clifford Neuman, director of USC Center for Computer System Security, states, “DETER allows us to model the internet so that we can understand how our systems will respond in the real world.”

Luke Berndt, a former Department of Homeland Security (DHS) system manager, adds, “It (DETERLab) lets us replay scenarios and attacks, looking at what configurations and solutions work the best against attackers.”

Over 200 organizations, 40 states, and 30 countries have made use of DETERLab and its tools for cybersecurity experimentation.

Educating future cybersecurity professionals

Besides being a controlled environment where researchers can test, DETERLab is an educational tool. Since DETERLab’s inception, 3,800 students have selected from 100 DETERLab-based classes offered at 40 institutions.

Not only students use DETERLab. The test bed affords academics a unique opportunity to share and verify. “I can release my data to scientists anywhere in the world, tell them to run the experiments and report what they get,” says Sandy Clark, a researcher at the University of Pennsylvania. “That is essential for peer review and validation.”

The future of DETER

The success of DETER has emboldened Benzel and her associates to map out what is needed to take a new bite out of internet crime. “Our vision for the future is a nationwide set of interconnected test beds that can coordinate operations and share data,” says Benzel. “And, we see DETER as providing both a cornerstone capability and a set of ‘lessons learned’ to help us move toward that vision.”

With Cybersecurity Experimentation of the Future (CEF), Benzel and her associates are turning their glimpse into the future into reality. CEF, a community-based study, is bringing experts and researchers from academia, industry, and government together to increase the effectiveness of cybersecurity by applying proven scientific methodology with particular emphasis on:

  • advancing the field of experimental methodology and techniques, with regards to complex systems and human-computer interactions;
  • sharing data to accelerate cross-organizational awareness and community building; and
  • improving experimentation infrastructure capabilities.

The group’s first step was publishing Cybersecurity Experimentation of the Future (CEF): Catalyzing a New Generation of Experimental Cybersecurity Research. The 164-page report provides an in-depth look at how to obtain the above goals, starting with the following.

  • Multidisciplinary experimentation: Cybersecurity testing must become multidisciplinary, adding engineering, mathematics, modeling, human behavior, sociology, economics, and education to the expected discipline of computer science.
  • Modeling the real world including human activity: Up until now, human activities and responses regarding cybersecurity have been for the most part guesses. The report suggests human interactions must be accounted for; otherwise, test results are invalid.
  • Open standards: The report stresses the need for open standards that all disciplines understand. “As a result, communities will be able to conduct, validate, integrate, and share experiments and results,” adds the report. “This fundamental ability is needed to enable broader research in cybersecurity, as opposed to working in narrow-sub disciplines.”
  • Reusable designs for science-based hypothesis testing: The design of test-bed experiments must reflect what is happening in the real world. The report also mentions, “Experiment designs should be validated and processed akin to the use of software development environments.”
  • Usable across disciplines: The report authors take into consideration that the test-bed experiments will not be only run by high-powered computer types. If they are, any advantage of involving diverse disciplines is lost.

The report suggests that going forward research infrastructure will be dynamic and applied multiple ways, thus how the technology is used and tested becomes an important consideration. To that end, the report suggests, “In the long term, we envision some degree of self-configuring or infrastructure-in-a-box capabilities to ease operational burdens, particularly for prospective researchers who are not from a traditional computer science background.”

Benzel cuts to the chase, saying, “To me, the point is that we, the good guys, need to be able to experiment with new ideas for cyber protection. This is particularly true when we are considering advanced research. It is not simply a case of pass fail testing but instead exploring what-if scenarios. This all leads to the need for a ‘science of cyber security experimentation.'”

Note: The primary source of funds for the DETER Project has been the DHS.