Not so long ago, one of my clients called me in for a
process and measurement readjustment project. We started with the three core
services—infrastructure, development, and customer service. The measurements we
selected covered a range of procedural, operational, and creative factors. The
metrics for each measurement emphasized behaviors we hoped would be more in
line with the company’s goals.

However, change by measurement takes time. While we waited
for our first batch of quarterly results, my client asked me to deal with one
of the most nonconventional IT services. The providers were well noted in the
company for their troglodyte-like behavior and lack of reliable measurements.

This team, the Security Services Organization, worked
primarily with the external Web services and the network switches to create
security schemes. In theory, they offered data security services to the entire
organization. In reality, data, functional, and application security devolved onto
other groups.

My client asked me to first define this group’s
responsibilities based on various organizational charter documents and their
current activities. Then, he wanted measurements and metrics to guide them back
into the organization. Given his level of disgust with their current behavior,
firing the entire team was an option on the table. However, he asked me to
exhaust all possible avenues before making that recommendation.

The current state of security

The security team, like many such groups at other clients’
companies, spent a great deal of time talking about potential security threats.
They researched threats on the Web, studied esoteric-seeming materials, and
made complex pronouncements from on high. Their presumed level of technical
expertise (whether earned or not) gave their statements inordinate weight with
the rest of the IT staff.

The security team focused on external threats, firewalls,
and occasionally VLAN segmentation. Their stated goals were to prevent an
external intrusion and internal assaults through the wireless network. They
played through elaborate scenarios in the test lab, exploring ever more exotic
forms of attack.

At the same time, the vast majority of the company’s
resources were inadequately secured. Customer support managed most of the data
security for the organization. The developers, adrift without clear direction from
the experts, gamely worked to plug holes in their applications. The server
administrators in infrastructure managed their own patch schedule, sometimes at
odds with one another.

The authorization assignment and verification process also
needed a great deal of work. Any user could call the help desk to get access to
nearly any function. With the conflicting and contrasting security schemes
produced by the developers, no one really had any idea what “appropriate”
security levels might be, or what functions should go to whom.

The latter might be considered a problem with the customer
support organization. In part, it was. At the same time, the responsibility for
auditing the system, establishing clear access levels, and monitoring the
process all logically devolved onto the security organization.

Measuring the security service

I sat down to design measurements and metrics. The team
needed a sharp kick in the pants to get moving in the right direction. With
that in mind, I intentionally installed measurements we would have to
deemphasize in the long run. The measurement categories I chose were audit
compliance, budget, communications, incident response, and process management.

Audit compliance caused the largest number of complaints
from all involved parties. We gave the security team a list of all the security
areas in their control. Each week, they were to return to us a list of the
activities they took in order to manage those areas. Each month, we asked an
independent group along with representatives from customer service,
development, and infrastructure to verify whether the work had the stated
effect. If it did, they received a pass on this measurement. Each failed item
reduced their potential score. Anything over a 20-percent failure caused them
to fail this measurement.

Budget compliance (a perennial favorite) allowed us to
measure the amount of money spent vs. the amount of money allocated for
security activity. We also added another metric: amount of money spent vs. the
percentage of high-to-medium risks (rather than threats) addressed. Failure to
meet high-level risks, or spending money/time on low-level risks/threats while
higher-level threats went unattended to, reduced their budget score.

Including communications as a measurement raised a host of
issues. The security team felt they were responsible only for technical
solutions. My client felt they were a key communicator in the overall system
management process. In order to move them more toward this latter role, we
implemented metrics measuring the time spent usefully participating in meetings;
the speed in responding to queries; and the clarity of communications regarding
specific security queries. We felt these measurements would, over time, create
a useless focus on sudden rather than meaningful communication. However, during
the first few measurement cycles, they might also help to focus the team on
proper communications techniques.

Incident response involved metrics sampling—the speed in responding
to security breaches, appropriate auditing of security changes, and the speed in
responding to security-related changes concerning the products in the
environment. If the team responded quickly, performed their auditing tasks in a
reasonable time frame, and worked with the other IT departments, they received
good marks. Each time they failed to involve others or delayed their response,
their score decreased.

Process management, as a measurement, was intended to drive
the team toward establishing and managing a comprehensive security plan. Here’s
where I disagreed with my client: I felt the plan’s creation should be a
project, while its maintenance could be a measurement. But he didn’t want to
divert the team’s attention. In the end, we settled on the following metrics:
successful audits of the IT security processes, dissemination of the IT
security processes, and adaptation of the IT security processes. Activity that
fell into these three categories increased the team’s score in this

We also disagreed on whether the team should automatically
fail if they did not take action in the process management category. My client
set the base score to automatically fail unless the team spent time actively
working in these areas. I argued that the team’s scores in other areas should
be weighed as well, since they indirectly indicate process awareness. However,
he felt that the threat of failure, and the attendant action, was important to
his overall goals.

With security, my client and I compromised on long-term
measurements in order to achieve short-term goals. Would you have made the same
decisions? Would you measure the same things? Why or why not? On what level do
you need your security team to deal with threat rather than risk? What other
issues might come up?