Leadership optimize

10 efficient ways to produce useful metrics

Accurate metrics are essential for assessing performance and making informed decisions, but companies often rely on flawed information that paints a misleading picture. Alan Norton explains the most efficient strategies for obtaining valid, useful metrics.

The one constant in my career has been the collection and distribution of company information. I didn't know the cost of the metrics I was producing. Those costs were never measured. But I did know my labor costs and the costs of the supplies I was purchasing. Multiply that by each department in the company where I worked and I knew that the numbers had to be getting real big, real fast. I also knew that there were opportunities there for big cost savings.

It is not difficult to make a case for more efficient metrics. Significant cost savings can be realized by reviewing and improving how and what metrics your company produces. Metrics are expensive. Add up the true total costs and they can be quite considerable:

  • Printers
  • Paper
  • Printing supplies
  • Labor
  • Development
  • Production
  • Distribution
  • Maintenance

And the costs don't end at document creation. Metrics are shared multiple times, either electronically or physically. The security of company sensitive metrics is an additional responsibility and cost. The metrics must be managed, stored, and retrieved --all activities more costly than you might guess.

I won't try to argue that metrics are in any way fun to discuss. However, their importance to the health of a company is undeniable. Get the metrics wrong and a company's full potential will not be realized. I have seen enough to know that the metrics used by companies both large and small can be improved. Here are 10 ways to do just that.

Note: This article is also available as a PDF download.

1: Measure performance at aggregate levels and exceptions at lower levels

I have produced a lot of charts in my time. In retrospect, I realize that I produced too many charts. Many of those charts provided no value at all -- they showed performance on schedule or close to on schedule for a number of subassemblies. Printing these charts might have given managers a reassuring sign that all was well, but little else. The better approach is to use exception reporting and set upper and lower limits. Sure, go ahead and measure performance at higher levels. But report only exceptional performance, good or bad, at the more granular levels.

2: Eliminate metrics with little or no value

Know the cost of each metric. Are all your metrics worth reporting? A cost benefit analysis can help identify reports that are costing more than they are worth. A best guess estimate of potential savings must be included in the analysis, so it is not as simple a process as it might first appear.

It can be difficult to discard those old, comfortable shoes when you're rummaging through your closet. But if they're no longer useful, they should go. The same applies to those charts and reports that managers have become comfortable seeing. Nevertheless, if they are of little or no value, they should go.

3: Avoid tying incentives to lower level metrics

Meeting goals at the department level can be detrimental to the company's overall performance. For example, a help desk manager might be paid a bonus for increasing completed tickets at tier one. But this is counterproductive if the overall costs rise due to the increased workload at tier two and lower. Bonuses and other incentives should be tied to the corporate bottom line and not departmental performance.

4: Avoid metrics that are inaccurate, misleading, or ambiguous

Producing metrics that are inaccurate or ambiguous can be costly to correct if they lead to bad decision making. There are a number of reasons why metrics can be inaccurate, misleading, or ambiguous:

  • Invalid assumptions
  • Errors in data collection
  • Incomplete data
  • Outdated data
  • Questionable/unverified data
  • Difficult to measure data
  • Subjective (not objective) data
  • Complex and multifaceted data

Some metrics are difficult if not impossible to measure. Managers are nevertheless tasked with boiling down metrics like software development into one or two charts. Similarly, other metrics, like employee happiness, just don't translate well into numbers and should be avoided. I'm not suggesting that measuring this type of information should be avoided altogether, just not presented as numbers in chart format.

5: Avoid skewing the data

One simple example of data skewing is reporting costs over a long period of time. 2010 dollars are not the same as 2005 dollars. If you are not compensating for inflation, you are delivering misleading information. Obtaining too small a sample or a non-random sample are other common ways that data can be skewed.

6: Standardize and consolidate

Reduce the amount of information generated by eliminating redundancies and standardizing where possible. Financial reports, like manpower metrics, can usually be centralized and standardized for all departments. There are those unique metrics that can't be standardized for all departments, but many of these can be standardized across divisions.

7: Use unbiased personnel or outsource connect

Being a reporting guy my entire career, it isn't surprising that I ran across a manager or two who wanted to game the system. There are countless ways to lie with numbers or, at a minimum, hide poor performance by the appropriate (or should I say inappropriate) manipulation of the data. One example I saw was to use fiscal YTD values instead of a 12-month moving window. The obvious problem with this is that when a new fiscal YTD starts you have no previous data points to show a trend. And that is exactly the goal. It's a great way to hide bad numbers. I reported directly to the manager asking for the fiscal YTD format, so I was in no position to question his motives. The inherent problem created with this type of working relationship is another reason why metrics production should be centralized and reported directly to upper management or outsourced.

8: Automate whenever possible

Gathering data by hand and entering numbers into a charting program is a labor intensive and costly endeavor. Since charts are often generated weekly or even daily, one or more people can spend most of their time preparing the charts. If the data is available online, the charts should be automated. If the data is not available online and must be collected by hand, consider developing systems that can collect the data at a reasonable cost.

9: Consider going paperless

If your company hasn't already done so, going paperless can save a lot of money. However, paper should still be an option and care should be taken before converting a report to online only. If it takes more time to look up a metric online, the additional labor costs can far outweigh the savings.

10: Be careful what you measure

All too often, overly simplistic metrics focus on only part of the whole picture. A metric like "IT spending as a percentage of revenue" treats IT services as a cost. But such incomplete information can lead to poor decision making. The challenge for IT managers is to develop metrics that show the value added by IT and get them under the CEO's nose.

The very act of measurement determines where managers will place time and resources -- often at the expense of other unmeasured metrics or other departments' metrics. Measure too little and important performance metrics can be missed. Measure too much and the focus can be lost from the mission-critical metrics. The costs of measuring the metrics can then quickly exceed the benefits. The metrics package should be carefully selected to balance these tradeoffs.

The bottom line

Metrics are like laws. They are born but never seem to die. An annual review of the company's metrics package provides a good opportunity to add new metrics, update out-of-date metrics, and discontinue those metrics that are no longer needed. Select your metrics package carefully. What you measure is what you get.

For further information, check out the whitepaper Efficient Metrics: Be Careful What You Measure.

About

Alan Norton began using PCs in 1981, when they were called microcomputers. He has worked at companies like Hughes Aircraft and CSC, where he developed client/server-based applications. Alan is currently semi-retired and starting a new career as a wri...

14 comments
jkameleon
jkameleon

Otherwise, it WILL be gamed. In order to be meaningful, metrics must involve all knowledge from the field it measures. That's why it's useful only for simple tasks, like shoveling coal. For creative, knowledge work and problem solving it's less than useless.

Alan Norton
Alan Norton

It's ironic but I have always tried to avoid "the reporting guy" role. But everyone needed charts, reports and financial information so from the beginning I was stuck with the role. I made the best of it by automating where I could and it paid the bills. I shouldn't complain since this was my way into MIS, as IT was called at the time. In my last position we designed and implemented some sophisticated automated report generation 'on the fly' and report specific data storage in databases. Perhaps this is why it has taken me nearly three years to get around to discussing metrics here at TechRepublic. Metrics are about as dry as accounting but important nevertheless. Getting the metrics right is a lot more difficult than it appears. As always, I will be popping in occasionally to answer any questions and add to the discussion when I have something intelligent to add.

Michael Kassner
Michael Kassner

Could you please provide your definition of metrics?

Alan Norton
Alan Norton

I have read that 'lines of code' is used to measure the performance of programmers. I can't think of a worse metric for measuring software development.

Alan Norton
Alan Norton

Hi Michael. Here is my definition: Metrics - A measurement or measurements of the performance or activities of any human endeavor, especially where groups of people are organized.

jkameleon
jkameleon

Quoting from memory, in chronological order (more or less). Metrics: Lines of codes. Result: A lot of empty lines & useless comments. Metrics: Noncomment lines/Size of compiled code. Result: A lot of copy/paste code reuse & functions that are never called. Metrics: Function points. Result: Unnecessary operations. Write data to disk, read from it again, and write, and read, and format, and parse, etc etc etc. Each operation counts as one function point. Metrics: Code quality, lack of bugs. Result: Nobody does nothing for fear of making an error. Metrics: Code quality, testers paid by the bug they discover. Result: Programmers deliberately make bugs, and report them to testers. Stimulation is split 50-50 between programmers & testers. Ever since the times of Frederick Taylor, every working stiff considers gaming the metrics the most sacred duty to God & humanity. This is especially true for programmers. The fact of the matter is, that measuring problem solving tasks makes no sense. The only way the problems can be measured is Kolgomorov Complexity: Solve the problem in optimal manner, and measure the size of the solution. That holds true for intellectual work in general. According to experience gained in British public health system, the consequences of metrics in medical field can be deadly.

Quasar Kid
Quasar Kid

Metrics??? Metrics??? Really??? Stop the obfuscation and say what you really mean. How about saying "We are going to measure your performance using quotas." I can't stand it when business people talk like government bureaucrats...

robert
robert

A key lesson to learn in any application of metrics (or management methods in general) is that of perverse incentives. Also sometimes known as the Law of Unintended Consequences. I suppose we have all seen countless instances of the same kinds of outcome.

Alan Norton
Alan Norton

That was very enlightening and entertaining. Was nothing learned in this process by the metric makers? ;-)

RudHud
RudHud

"Metric" always struck me, too, as a useless neologism for "measure" -- so I looked it up (etymonline.com). They cite it in its current use back to ancient Greece, and in English to the 14th century. The Greek use, based on the meter of a poem, traces to the Indo-European root *me, meaning measure. Neolithic herders kept metrics on proto-cattle herd sizes. So no, "metric" for "a type of measurement" is not a neologism. As for definition, Microstrategy manuals say that if you can do arithmetic with it, it's a metric. That's a pretty good description, but only if you add that you can, with a near-trivial conversion, do arithmetic on anything.

Alan Norton
Alan Norton

If not already, metrics should be centralized and owned by IT.