CXO

How to avoid misguided metrics

Times change and so do benchmarks. Here's what to keep in mind when a new benchmark appears on the horizon.

Image: iStock/mel-nik

As part of a relaunch of my foray into endurance sports, I recently purchased a new road racing bicycle for the first time in about 15 years. Those unfamiliar with cycling might assume there has been little change in a machine that's been with us for nearly 200 years; however, what I found most interesting was a change in one of the key metrics used to describe the performance of a bicycle.


When I bought my last road racing bike, cyclists were obsessed with the weight of the bicycle and each component attached to it, to the point that many cyclists became self-described "weight weenies." The logic was that weight savings translated directly into speed during climbing; logically the less weight carried up a hill the faster one could climb that hill.

Fast forward to the present, and aerodynamics have now supplanted weight as the holy grail of bicycle and component performance measurement. Instead of weight, cycling forums are filled with discussions about watts, representing the effort saved by a more aerodynamically-efficient bicycle component. Yaw angles, wind tunnel tests, and drag coefficients are the new currency of the realm, and cyclists spend endless hours discussing whether one handlebar can save a watt or two (less than a half-percentage efficiency improvement for the average cyclist) versus another.

Misguided metrics

IT has experienced similar evolutions in the benchmarks it uses to track performance. Around the time cyclists were obsessing over screws made from exotic metals in the name of saving a few grams, IT leaders were obsessing over Total Cost of Ownership (TCO), a metric designed to account for direct and indirect costs of IT savings. IT vendors would extol reducing TCO as a gold standard for IT management; however, like the cyclist obsessing over a gram or two rather than losing a kilo of excess weight, regarding only the cost aspect of IT could often cause leaders to overlook the larger picture.

Additionally, like the cycling spouse who grows tired of hearing about their partner's expensive new gadget purchased in the name of saving watts, so, too, did non-IT leaders tire of hearing about difficult to track TCO savings, and tended to focus solely on the costs of those transactions.

Bicycle companies eventually realized that weight savings did not make for a very interesting story for cyclists who were not interested solely in climbing, and also needed a new story to sell to those who had invested in the latest and lightest technologies. Similarly, IT vendors realized that cost was not very compelling outside IT and finance circles, and both seemed to search for a new metric at a similar time.

It's interesting to note that in each case, the metric was easily measured and objectively comparable. Just as one wheel can clearly be lighter than another wheel, one software package might cost less than another. However, as you try to marry benefit to these metrics, they begin to fall apart. Will a 20-gram savings make a cyclist faster? Will a 20 percent reduction in TCO matter if the technology isn't used by the community it's meant to serve?

Checking for fitness and function of the metric

Saving weight or cost is rarely a goal in itself, hence the demand for more sophisticated metrics that actually map toward an end goal. IT realized this and started trying to quantify return on investment (ROI), or the more mercurial business value, in an attempt to show that technology could actually contribute to the bottom line. In some cases, this correlation is readily identified and observed, but in many cases the connection is more tenuous and the metric itself must be validated. A sales tool that gives salespeople five extra hours each week could readily be linked to increased sales, but an advanced analytics tool might have a less obvious connection to the bottom line.

In these cases, we can build a series of hypotheses that are then tested, primarily attempting to identify fitness and function. Just as the cyclist assessing whether a new aerodynamic bicycle will help their cycling might ask whether they're fit enough to take advantage of the tool, so, too, should an IT organization honestly assess whether it has the sophistication and organizational fitness to achieve a hypothetical benefit. If the fitness exists, is the tool being evaluated appropriately functional? Our theoretical cyclist might determine that the new bicycle is most beneficial on steep descents. If he or she lives in a flat part of the country, an aero-focused bicycle will accomplish little, just as a new technology that pushes a metric that fails the functional test will do little to advance the objectives of your IT organization.

Metrics don't always evolve to the benefit of every potential consumer, and can be driven more by trends or vendors attempting to excite consumers than stated need. Avoid becoming the IT equivalent of the weight/watt-weenie and use the metrics that are most relevant for the technology challenge at hand.

Also see

About Patrick Gray

Patrick Gray works for a global Fortune 500 consulting and IT services company and is the author of Breakthrough IT: Supercharging Organizational Value through Technology as well as the companion e-book The Breakthrough CIO's Companion. He has spent ...

Editor's Picks

Free Newsletters, In your Inbox