Data Centers

Benchmarking requires a common set of processes and metrics

There is more to benchmarking than meets the eye. Gathering meaningful data for a benchmarking project requires careful thought about how common processes and metrics can be accumulated. This article lays out some steps to consider.

You might think that benchmarking is easy. All you have to do is ask companies to report a common set of metrics and then compare them to see who is doing the best. For instance, let’s say that you ask a number of companies how much money they spend on “support” in a year. Your assumption might be that the company that spends the least on support probably has the most efficient support organization. But this is not a reasonable assumption. Think of the variations you will receive:
  • Different organizations are different sizes, which will automatically result in higher and lower support budgets.
  • Different organizations have different definitions of support. Some organizations have support organizations that only do true support. In other organizations, the staff works on enhancements and projects as well.
  • It’s not clear what the scope is. Are you just interested in business application support, or will you also include network administration, the Help Desk, telecommunications, and so on?

Since the request for information is vague, it will be impossible to compare companies and the overall benchmark is worthless. Let's look at ways to garner more meaningful data.

Earlier articles on benchmarking
This article is the third of four in a series on benchmarking. To read the previous two, click on the following links:

Benchmarks can be misleading
I have participated in simple benchmarking efforts where it was not always clear what information was being requested. For instance, one benchmarking questionnaire asked how many hours my company spent on testing. Some companies have standalone testing groups, but mine did not. So, I had to estimate the amount of time the developers might spend in the testing function. Since I’m sure all of the participating companies struggled in similar ways, I think the resulting value of the benchmarking study was marginal at best, and misleading at the worst.

Setting up a good benchmark program
To avoid such unclear questions, responses, and results, a good benchmarking program will have the following characteristics:
  • A scalable approach for companies of all sizes, including medium and small organizations. Most of the information needs to be normalized in such a way that it allows comparisons between companies and between projects.
  • Standardized process definitions and metrics to be used across all industries and companies. You want to make sure that all participating companies can map their internal processes into the benchmarking processes so that valid comparisons can be made.
  • Clear demographics for comparing apples with apples. Demographics are descriptive characteristics that allow you to understand more about the companies, organizations, and projects that you are comparing. Company demographics, for instance, allow you to compare yourself against companies in similar industries or similar sizes. Project demographics allow you to compare yourself against similar projects. For instance, you might want to compare how you specifically deliver Web applications vs. how other companies in your industry do so. Or, you might want to compare how you deliver traditional client-server projects against other companies that are using more agile techniques.
  • Emphasis on simplicity, practicality, and the cost of data collection. The harder it is to collect the data, the less likely the information will be accurate.

It might be hard to visualize how the benchmarks are set up. So let’s run through a simple example of benchmarking how companies deliver business applications.

First, define the common processes
I’ll define the first two steps in a project life cycle as "Planning" and "Analysis." Your life cycle might be different, but I am attempting to come up with a common process that each company can map their processes into. For each step, you would need to generate information such as a description of the process, the main customer(s) involved, and the person(s) who execute the process. In addition, it would help to know the events that must happen before this process can occur, key inputs to the process and where they come from, key outputs of the process and where they go, and how you know when the process is complete.

This type of information will allow each company to map the steps in its internal methodology against the standard processes in the benchmarking study. It is also likely that each of the processes will be broken down into further sub-processes. If so, each sub-process will also need a common definition.

Next, define the common metrics
Once you have a set of commonly understood processes and sub-processes, you can define a set of metrics that represent various aspects of the workflow. The metrics should be easy (or relatively easy) to collect and they should represent some fundamental piece of data that can be used to create the benchmarks. The metrics themselves cannot be compared from project to project, or company to company. Only when the metrics are used in the benchmarks can the comparisons be made. Examples of metrics for the Planning and Analysis processes might be:
  • Estimated and actual effort to complete
  • Estimated and actual duration to complete
  • Estimated and actual cost to complete
  • Number of defects

For each metric, you must also provide a common definition, including how the metric is calculated, the frequency with which it is collected, when it is collected, and some categorization data that allows you to collect a well-rounded set of metrics.

Lastly, create the benchmarks
The metrics are combined into benchmarks that allow companies to compare themselves with other companies. Examples of benchmarks include:
  • Project Duration Variance (actual duration / estimated duration)
  • Project Effort Variance (actual effort / estimated effort)
  • Project Cost Variance (actual cost / estimated cost)

You also need to define the benchmarks, including how the benchmark is calculated based on the metrics, benchmark classification (to create a balanced set of benchmarks such as financial, client satisfaction, quality, etc.), and how to interpret the resulting benchmark numbers.

In the example above, for instance, you might say that each company should strive to have a Project Duration Variance of 1.0, which would mean that the actual duration of each project was exactly as estimated. If your company ranked 15 out of 20 companies in the benchmarking study, it would mean that 14 companies deliver against expectations better then you. It doesn’t mean they deliver cheaper or faster then you. Those would be different benchmarks. This benchmark only tells you how you deliver against your estimates.

The story behind the numbers
Benchmarking requires much more than asking companies to report numbers. You first need to establish a common benchmarking model that all companies can understand and use to map their own processes to. Then you can set up metrics for each company to report and benchmarks that each company can use for comparison.

In the next and last column of this series, I will look at how companies gain value from the benchmarking program. As you will see, the real benefit of benchmarking is what you do with the information.

Editor's Picks