Leadership

Agree on balanced metrics to measure project success

Project management guru Tom Mochal advises that you and your client agree on the right metrics before the start of a project so that clients will get what they want.
Editor's note: This article was originally published October 29, 2001.

The dilemma

Mary was assigned to a project for the Finance Division when the prior project manager resigned. It was a tough spot, but Mary did an admirable job in bringing the project to completion. Although the solution was ultimately implemented, some question remained as to whether the project was successful.

I attended the project conclusion meeting, and I could tell it did not go as Mary had expected. Mary came prepared with a set of metrics, but the business client did not accept them at face value.

"Mary, it appeared that there was a difference of opinion on what the facts were on this project," I began. "You attempted to initiate a fact-based discussion. Why do you think it didn't work out that way?"

"There was a lot of emotion built up over the course of the project," Mary replied. "I came into the project late and didn't realize the level of dissatisfaction some of the people felt."

"It sounded like they were also challenging the validity of some of your numbers and whether they were relevant," I noted. "For example, you said that the project completed on schedule, but the client said that the solution was implemented without adequate testing."

"That may or may not be the case," Mary replied. "We all agreed that we would implement now and fix any problems going forward."

"Yes, but the client said they were pushed into that decision because they could not afford to miss this monthly financial close cycle," I said.

I used this example to voice my overall advice for the future. "Mary, you have the right idea about project metrics. If you measure how you are doing, you will be in a much better position to have a fact-based discussion about the overall project success or failure. But you've missed a couple of important points."

Mentor advice

It's important here to distinguish between facts and statistics. Statistics, or metrics, can be gathered on myriad combinations of project and solution characteristics.

But be careful when using them to declare the success or failure of your project. First, make sure you have an agreement with the client on the set of metrics used to declare success. Secondly, make sure the metrics are balanced and broad enough to represent the reality of the project experience.

If there is a disagreement on the metrics gathered and what they mean, you will be challenged right away. You also run the risk of losing all credibility if you attempt to show that a project was successful by displaying only a narrow set of favorable metrics.

That's what happened to Mary during her project conclusion meeting. Her business client is not happy with the way the project was run, and they are not happy with the solution that was implemented. So, not surprisingly, they don't want to consider a set of metrics that indicates that the project was a success. A wider range of metrics would include:

  • Actual costs expended vs. the original budget and actual delivery date vs. the original schedule.
  • Quantitative metrics that describe the solution's performance, including response time and defects.
  • Qualitative metrics that describe client satisfaction with the solution, including ease of use, look and feel, etc.
  • Satisfaction feedback that describes the client's satisfaction with how the project team performed, including how quickly the team responded to problems, how well they communicated, how well they partnered, etc.

Mary should have proposed a set of rounded metrics that covered cost, schedule, performance of the solution, and performance of the team. She should then make sure there was an agreement with the project sponsor on how to use the metrics to determine how successful the project was.

Get weekly PM tips in your inbox TechRepublic's IT Project Management newsletter, delivered on Wednesday, offers tips to help keep project managers and their teams on track. Automatically sign up today!
11 comments
Steve Romero
Steve Romero

Another spirited conversation about metrics and measures. It provides, yet again, a great example of the difficult and controversial nature of the discipline. I am a proponent of metrics and measures - when they are meaningful. I offer the following tips to make them so: First and foremost, everyone must understand why we even care about the data good metrics provide: to make decisions. We use the data to determine: - Was my decision realized? - Was my decision good? - How can I make better decisions? Here are some good criteria to determine if your metrics are producing meaningful data: - People understand it - People know what to do with it - People want it The following are some elements of good metrics: IMPORTANT - Reflects the ultimate goals and purpose of the organization CONTROLLABLE - Is something that employees can directly influence ACCURATE - Reliably expresses what is being measured OBJECTIVE - Not subject to dispute EASY- Not burdensome or expensive to obtain TIMELY - Is available in time to make a difference COMPREHENSIBLE - Easily communicated and readily understood HARMLESS - Does not induce dysfunctional behavior And I will close with one final thought: If you are collecting data, and you don???t know exactly what decision is associated with the data, stop collecting the data. It is a waste of time. Steve Romero, IT Governance Evangelist http://community.ca.com/blogs/theitgovernanceevangelist/

Tony Hopkinson
Tony Hopkinson

no measurements no matter how great will convince an unhappy client. Unlike Mr Chameleon whose attitude I completely understand I do feel somethings can be done, and there is some value in them BUT in my experience metrics, far too often become more important than what they are measuring. I always remember that number of lines of code written was a popular one. I'm not sure that many of them are much better. And then there is the perennial problem of PMs simply reporting failure as opposed to managing success......

Daniel_J-22521388082541631442670333001745
Daniel_J-22521388082541631442670333001745

"Quantitative metrics that describe the solution???s performance, including response time and defects." I usually stick right to the meat and potatoes, which is aligning the functionality of the finished product with the original business requirements of the customer. I would like to do more in this area, and it seems like this article is implying that there is more to do... Software is my game, any suggestions for effective metrics would be appreciated. Any book suggestions would also be welcome.

jkameleon
jkameleon

Knowledge work, especially work which involves problem solving can not be measured. Period. Get over it.

jkameleon
jkameleon

The only feasible measurement of software is its market value, or productivity gain measured in $$$. _____ Since there is a lot of problem solving involved, software development is very risky undertaking. This risk has to be managed, not shifted around through metrics and similar tricks. Yea, tricks. As I said before, any measurement, which ultimately translates to money, must be exact. If it's not, it's fundamentally unfair, it's like inaccurate weighting scale on food market.

jkameleon
jkameleon

Every software metrics so far failed. Lines of code metrics results in bloated programs that do nothing. Function points results in bloated programs that do useless stuff. Number of bugs found and fixed results in bug market between developers and testers. Any metrics or measurement that ultimately translates to money must be exact. The only exact measure of software is Kolmogorov Complexity. It's obtained by finding the shortest, optimal solution to the problem. In order to measure the developer productivity exactly, manager would have to write software by himself, in the most effective manner possible, and compare. That, of course, makes not sense.

dobrien
dobrien

I think you probably belong to that group of professionals who give the rest of us a bad name. How many times have I heard "You can't measure our performance: it's much too complex and you wouldn't understand". Are you telling me that it's impossible to gauge whether or not the project team are doing a good job? Or maybe I'm missing the point: development/ implementation is not science after all, but magic?? As the title says, if you don't want to see....

CultOfTech
CultOfTech

.. which kind of amply illustrates not only your apparent ignorance of the scientific realm of project management, but also your ill-disguised manners when posting in a niche, specialized blog! :-)

Tony Hopkinson
Tony Hopkinson

There are some things that have to be done, by a certain time. All else is bollocks to make management look necessary. Short of spending a lot of time doing the work, so you can say how long it's going to take and decide whether you can do it :p , it's just a guess. If the guess was wrong, that means the guess failed not the project.

jkameleon
jkameleon

... but that's OK. I consider keeping one's mind closed to newage psychotherapies, bioenergetic healing, scientology, feng shui, project metrics, team building, motivation speaking, and similar management fads a good thing.

Mark Johnson
Mark Johnson

The article talks about metrics not measurements. The old analog (dial) car fuel gauge was a metric; it didn't tell you how many litres of fuel you had left but it did indicate when the tank was getting empty so you could choose to fill up. And you all know the consequences of ignoring that particular 'metric'. Agreeing project success (and failure) metrics are one of the most powerful ways to ensure that you understand your stakeholders; and if you report them regularly to the Steering Board then you should have powerful allies if it turns out that there were hidden agendas that undermined the project.