CXO

Structured and unstructured approaches to starting a metrics program

Starting a metrics-collecting program can be daunting if you've never done it before. Here are a couple of suggestions, one formal and one informal, for doing it right.


Question
I work in an IS department with about 120 people. We currently don’t collect metrics on any of our work. My supervisor wants me to come up with a few general metrics that will encompass only nonopinionated information, such as cost, duration, and number of help desk calls.

Do you have any suggestions on how we should start this program and what types of metrics to collect?

Answer
Do I have any suggestions? Of course! Your question states that you don’t collect any kind of metrics, but I would have to assume that you at least track cost metrics at an organizational level. All departments have a budget and usually have some way to track their actual expenses against the budget.

I know of two approaches for organizations looking to start collecting metrics. One is a formal, structured approach and one is an unstructured “let’s just do it” approach.

The structured approach
One way to get a metrics program started is to get a set of key stakeholders together and go through an exercise to set a formal process in place. The overall steps would include:

Identify criteria for success
First, you need to define what success means to your organization. You would normally look at your business plan, strategy, vision, departmental objectives, and so on. If you have no guidance at your department level, see if any of these documents exist at a division or company level. If you have no guidance at all, the group will need to spend some time identifying a candidate set of criteria that would signify success for the organization.

Assign potential metrics
Identify potential metrics for each of your criteria that indicate whether you’re achieving success. This is a brainstorming exercise, so you should identify as many potential metrics as possible.

Look for a balance
The potential list of metrics should be placed into categories to make sure that they provide a balanced view of the organization. For instance, you don’t want to end up with only a set of financial metrics, even though they might be the easiest to obtain. In general, look for metrics that provide information in areas such as cost, effort hours, project success, quality, productivity, client satisfaction, business value, and so on.

Prioritize the balanced list of metrics
Depending on how many metrics you have identified, prioritize the list to include only those that have the fewest costs to collect and provide the most value to the organization. If this is your first real effort to collect metrics, you'll probably want to capture a minimum core set.

Set targets
The raw metrics may be of some interest, but the measure of success comes from comparing your actual results against a predefined target.

Collect and analyze the information
Now comes the hard part. Set up the processes to collect the metrics and analyze them on an ongoing basis.

The unstructured approach
There’s another approach that I’ve seen work. The basic philosophy is “just collect something, even if it’s wrong.” In this approach, some key people in the organization get together and look for information that can be easily captured and from which certain aspects of success can be inferred.

This isn’t as bad as it sounds. You basically look for metrics that can be captured easily and start to capture and analyze them. After you collect the data over time, you get a sense for whether the metrics are providing value and whether you need to find more or different ones. This approach gets you into the habit of collecting and analyzing metrics first and allows you to improve your metrics over time. I’ve recommended this approach to organizations and projects having difficulty working through the more structured approach I described previously.

Your metrics should include client satisfaction
Your manager asked you to recommend a set of nonopinionated metrics. Obviously, your manager is putting a lot of emphasis on objective metrics that speak for themselves rather than subjective metrics that leave open the chance for bias. Capturing cost metrics is an example of objective metrics. Counting the number of help desk calls is also fairly objective, once you define exactly what a “call” is.

However, surveys can provide the fastest and cheapest way to gather good information when you are starting. After all, all the metrics in the world won't make you look successful if your customer has the perception that you’re not. Simple surveys can be used as a substitute for the hard, objective metrics. For instance, it may take a lot of work to determine the time required to resolve help desk tickets by severity level. On the other hand, you could send out a simple survey to some portion of people who submit problems to the help desk. One of the questions would ask users how satisfied they were with the time it took to resolve their calls. The feedback to this survey question probably goes a lot further to help you understand whether you’re meeting expectations, instead of just relying on the length of time for resolving a problem ticket.

Summary
I applaud you and your manager for trying to implement a metrics program. You need to capture metrics that give you some indication of whether (or how well) you’re meeting your organizational objectives and expectations. If this isn’t easy to determine, then look around for as many metrics as you can find that are easy to collect and provide some indication of your work. It sounds like you’re starting small, but it may be too small. I think you can collect a better set of metrics without too much pain.

I think you also need to include some survey, “opinionated” metrics, especially when you’re just starting out. Survey metrics can show how well you’re meeting expectations, what level of quality you’re delivering, and how easy your group is to work with. Later, you may be able to find more objective metrics to utilize in some of these areas. However, at this point you can get more value from gathering the perception of your client, analyzing the feedback, and improving your processes over time. Then continue the measurement process to ensure you continue to move in the right direction.

Editor's Picks