Reported data loss due to security breaches is not slowing down in the least bit, as the graph below (courtesy of DataLossDB.org)
vividly points out.  What’s more, these statistics only include
publicly reported breaches. One can only imagine how many security
breaches are unreported by organizations wanting to avoid public
scrutiny.

And did you notice what happened during 2009
— interesting isn’t it? I’m told there were several reasons for the
drop in reported data-loss events. The one being championed by most was
the introduction of security metrics. It seems security metrics as a
tool started gaining real credibility around that time. The SANS
Institute paper, Gathering Security Metrics and Reaping the Rewards, released during 2009 mentions:

“Many
substantial benefits can be derived from initiating a security metrics
program, and there is little reason for delay. At the onset it requires
only a meager investment comprised mostly of the time spent planning,
gathering data, and producing each report. This makes a security metrics
program an intriguing project, especially in economically challenging
times when funding can be tricky to secure.”

What are security metrics?

Security
metrics are often misunderstood, being referred to as a measuring
process, and that is not the case. Shirley C. Payne in her SANS
Institute paper, A Guide to Security Metrics, explains the difference:

Measurements provide single-point-in-time views of specific, discrete factors, while metrics are derived by comparing, to a predetermined baseline, two or more measurements taken over time. Measurements are generated by counting; metrics are generated from analysis. In other words, measurements are objective raw data, and metrics are either objective or subjective human interpretations of those data.

Next, Shirley describes what would be considered a “useful” metric:

“Truly
useful metrics indicate the degree to which security goals, such as
data confidentiality, are being met, and they drive actions taken to
improve an organization’s overall security program.”

It’s hard to
argue with the DataLossDB.org graph and what experts were saying.
Security metrics are doing their part to reduce data loss.

Number of data-loss events started increasing again

So
what happened in 2010 and afterward? Why did the number of data-loss
events trend upward each year? Security metrics were supposed to help.
Security metrics did one thing for sure, convince upper management to
spend money on new security programs as reported by tech media and
analysts alike — Gartner, for example:

“While
the global economic slowdown has been putting pressure on IT budgets,
security is expected to remain a priority through 2016, according to
Gartner, Inc. Worldwide spending on security is expected to rise to $60
billion in 2012, up 8.4 percent from $55 billion in 2011. Gartner
expects this trajectory to continue, reaching $86 billion in 2016.”

We
have security metrics in place, companies are spending gobs of money on
security, and I don’t think companies are reporting security breaches
for the fun of it. What’s going on?

Once again, there is no
simple answer. How could there be, just think of all the different
variables that come into play. But since we’re discussing security
metrics, let’s see if this reporting methodology has anything to do with
it.

New report may have the answer

While searching for information on security metrics, I came across the article, Are Security Metrics too Complicated for Management?, by Shelley Boose
of Tripwire. In the piece, Shelley mentioned a survey made by the
Ponemon Institute. Commenting on the survey, Dr. Larry Ponemon, chairman
and founder of the Ponemon Institute, said:

“Even though most
organizations rely on metrics for operational improvement in IT, more
than half of IT professionals appear to be concerned about their ability
to use metrics to communicate effectively with senior executives about
security.”

That was enough for me. I contacted Shelley, asking for more information about the survey; she referred me to this Ponemon report. Of immediate interest to me was the section about security metrics (PDF download). It started off with a bang: “Security metrics — important, but still not effective for communicating risk.”

It seems I may have found at least one reason why data-loss events are still on the increase.

Survey methodology

One
thing I always appreciate about Ponemon Institute is their willingness
to publicize the methodology they used in surveys, this one being no
exception (PDF download):

“A
sampling frame of 24,550 individuals of U.S. organizations and 18,012
of U.K. individuals who work in IT operations, IT security, business
operations, compliance/internal audit and enterprise risk management
were selected for this survey.”

42,562 individuals surveyed should give provide a decent sampling. Now let’s get to those survey questions.

Survey says

Question
1
: How important are metrics in achieving a mature risk-based security
management process? (Blue responses are from the U.S and red responses
are from the U.K. in all of the graphs.)

A large majority of the respondents did not dispute the importance of security metrics as a key performance indicator.

Question 2: Do you believe that your company’s existing metrics are properly aligned with business objectives?

The
response to this question may give us a glimpse at why security metrics
are not working correctly. Over 50 percent of the respondents either
said “no or were unsure” if security metrics aligned with business
objectives. The report ventures a guess as to why: “One potential
contributing factor in this disconnect is that security professionals
have traditionally viewed metrics as valuable operational performance
measurements, while executives tend to evaluate security based on cost.
Neither of these approaches is well adapted to communicating the
effectiveness of risk-based security programs.”

Question 3: Please
rate your effectiveness in communicating all relevant facts about the
state of security risk to senior executives.

Another hint at why
security metrics may not be working, close to 50 percent of the
respondents feel they are not effective in communicating the state of
security risks to upper management. The next question goes to the heart
of the matter, asking why the respondents do not create metrics that are
understood. The answers may surprise you.

Question 4:  If no or unsure, why? In other words, why don’t you create metrics that are well understood by senior executives?

The
fact that close to 50 percent of the respondents felt information
provided by security metrics was too technical, or that other
departmental issues took precedence was enough for those editing the
report to comment:

“In the same way it’s not acceptable for CFOs
to say they’re too busy to prepare financial reports for the board or
senior executive team, in the near future it will not be acceptable for
senior IT leaders to be too busy to prepare understandable security
reports. Security professionals must find or create metrics that are
more broadly understood by business leaders.”

Final thoughts

The
survey appears to have found the disconnect –business speak versus IT
speak. Business metrics that executives are familiar with tend to
reflect strategic goals, prioritizing cost over less tangible security
benefits. Where security metrics favor operational goals, and prioritize
technical improvements over business contingencies.

I’ll let the report have the last say:

Finding
meaningful ways to successfully bridge this communication gap is
critical to broader adoption of risk-based security programs. The onus
for this effort clearly lies with IT security and risk professionals.