Some managers love to look solely at numbers when assessing performance, and key performance indicators (KPIs) are right up their street. But how useful are they in the context of software development?

Strictly speaking, KPIs should not be confused with metrics in general. The term should be reserved for factors that really are essential for an organisation’s success, and that can be directly influenced by the individual or group being evaluated. Just because some aspect of a job can be measured, it is not necessarily worth measuring.

While it may be tempting to regard simple measures such as the number of lines of code written and debugged per day as KPIs, that could be seen as an early twentieth century perspective that focuses on efficiency rather than effectiveness. In most cases, developers are not the contemporary equivalent of stevedores unloading ships by hand. The role of the modern Australian developer typically has a creative side that is not amenable to such mechanical measurements.

“It’s not practical to have KPIs for software developers,” said Guy Harrison, chief architect for database solutions at Quest Software. “We don’t have any form of KPIs … for individuals.”

The best developers are more like craftsmen than process workers, Harrison said. Creativity is needed to produce highly desirable software, and there is no good way to measure that. If you rated the best developers on function points per day, for example, they would probably get a low score, largely because they are likely to be assigned the most difficult parts of the project.

One problem with KPIs is that they can be self-defeating if they are not truly aligned with overall business objectives. Developers are smart, said Harrison, and will optimise their efforts to whatever you measure. That probably will not give the best outcome if you measure the wrong thing. A key goal is a gelled, highly productive team, and that involves helping each other and sometimes working on foundations that are not amenable to measurement. Unsophisticated measures of individual output may work against that.

Around 10 years ago, software metrics were a hot topic and much effort went into finding which were important, but it turned out that none were, said Harrison. It might be appropriate to use KPIs for routine work such as report writing, but successful software companies are “trying to foster collaboration and excellence.”

Shanker Krishnan, general manager, products and services at Mastersoft, has racked up over 20 years in the industry and believes metrics such as function point analysis and lines of code might have been worthwhile in the COBOL days, but are less suitable today.

Owen Batt, technical director and co-founder of Smartpath said, “it’s not a smart move” to use KPIs such as lines of code, as it is better to use fewer lines to achieve the same end.

However, “all of our staff members would have some sort of KPI” with associated bonuses or other benefits, he said, but they relate to longer-term targets such as completing a module within an agreed timeframe rather than day-to-day considerations.

Traditional KPIs are not relevant, agreed Greg Brady, general manager operations at VeCommerce. “They’re probably not leading to the objective of reaching the [desired] outcomes of the business” and they tend to squash innovation and creativity.

Yet he is a believer in KPIs, as long as they are the right ones. VeCommerce — which develops speech and telephony software — focuses on customer benefit and business outcomes, and therefore sets KPIs accordingly. They are negotiated with the customer, and then rolled down to development teams and other groups within the company. A prime example is meeting completion dates, with the opportunity to earn “early achievement bonuses.”

Another involves the degree of automation delivered by the products, which is important to VeCommerce’s customers. An organisational-level KPI might be that a certain percentage of calls should be handled by the self-service system without resorting to a contact centre agent, and that trickles down to developer KPIs.

The idea of a “software factory” that is highly process driven and amenable to simple metrics is probably appropriate for maintenance and other predictable situations, Krishnan observes, but without predictability there can be no benchmarks, rendering KPIs meaningless.

Straughan Schofield, Tower Software’s general manager, product development and support, made the converse point, suggesting that the “management by observation” approach he expects from team leaders and senior developers would be inappropriate for highly specified projects where staff have little or no opportunity to innovate. “I don’t think this would work at all.”

US and European organisations tend to adopt formulaic approaches simply because “that’s the way you do it,” he said, whereas Australian companies take a more pragmatic approach.

When Tower’s CEO wanted to introduce KPIs across the organisation, Schofield resisted their application to his group. Software development is about innovation, he explains, and that requires the freedom to follow opportunities as they arise. It would be counterproductive to give incentives that work against that principle.

One suggestion was that the QA team could be judged on the number of units of work that they signed off, but Schofield argues that quality testing is more important than raw volume, and he wants QA staff to think about what they should be doing to deliver value to the organisation and its customers.

In place of KPIs, he favours formal lines of communication and administration, with (for example) team leaders and senior developers expected to know who is performing to the expected level.

Ben Wells, chief technical officer of Altium is another executive who questions the applicability of KPIs to the development process.

“At Altium we create complex, highly technical software. Of all the challenges we face, the most difficult is finding familiar metaphors, and designing cool GUIs around these metaphors, for complex technical procedures to make our software user friendly and intuitive,” he said.

Altium regards this as part of the developer’s job. “Our developers are not given specifications but are given projects characterised by high level problems which our users face or opportunities we can give them,” said Wells.

“Our best developers will naturally focus on this level, and will often experiment with the software — over many iterations — to find something which meets all of the objectives. I cannot think of any KPI which could measure performance in this area: creativity and ideas are impossible to quantify.”

Deadlines

On-time completion is the KPI most commonly mentioned by the executives we interviewed.

Schofield said his developers are expected to estimate how long a job will take, and then they are judged by how well they stick to it. This is important because the rest of the company works around the output of the development team.

While you might expect people to pad their estimates to provide some wriggle room, “most developers are show-offs” and are more likely to underestimate, he said. “People really want to do a good day’s work.”

Consequently, it is up to managers to ensure that all the necessary work has been factored in, not just the interesting pieces. Once a realistic schedule has been set, “finishing early is seen … as inaccurate as underestimating [the time needed],” said Schofield, as it can result in a product being later to market than necessary as marketing and other associated activities are planned around the expected completion date.

Quest tracks the estimated time to complete a unit of work against reality, said Harrison, and the variance is fed back into future estimate “in a non-judgemental way”. The best developers may take the longest and have the worst estimates because they work on the parts of the project that require the most creativity and are therefore the least predictable.

Krishnan summarises this by saying what is important is to understand individual and team performance, and to use that to plan well.

People management

The pattern of six-monthly reviews as used at Mastersoft does not mean managers can adopt a set-and-forget mentality between cycles: Krishnan often runs monthly informal reviews to make sure people are clear about what they are supposed to be doing and that their efforts are on track. This does increase the administrative load, so it is not always possible to go into detail on every point with every person.

The best developers are extremely productive because they understand the coding process, can abstractly visualise the problem, and they have domain knowledge that allows them to understand what users need. Combining these skills between one pair of ears eliminates the communication problems that are otherwise likely between different functions, said Krishnan, and so the best developers are “many times” more productive than their lesser counterparts.

“We have no doubt who our best developers are — it’s very apparent,” said Harrison, so he believes further metrics are unnecessary. The best developers are five to 10 times more productive than their average to low counterparts, he said.

The company tends to look for empirical data on employee performance only after an individual is judged to be a problem, for example when they persistently miss bugs that someone else finds. This is partly for legal reasons, as it is necessary to show the person concerned what they need to do to reach a satisfactory level before they can be dismissed for poor performance.

But the real goal is to create an environment where top developers can flourish and build great products. Allowing everyone to reach his or her potential is not done through measurement, said Harrison.

Managers need to “look after the talent” and put up with a degree of “attitude” if necessary, said Krishnan. It should be possible to cope with one or two prima donnas in a team. Unfortunately, there is a tendency for the most productive developers to be egotistical, dogmatic and with little respect for their colleagues.

“For cutting edge development you need people like that,” he said, adding that while they are valuable, they do need careful management. Organisations certified to CMM level 5 and similar standards do not want bright people like these, as they tend not to follow processes.

Small teams

Like Harrison, Brady favours small groups of skilled developers, which he calls “virtuoso teams.” They typically comprise four or five handpicked developers plus a project manager. The use of virtuoso teams is important for high-risk (ie, fast and difficult) projects, he said.

A team focus is important, so KPIs or metrics should reflect that, rather than concentrating on individual activity, said Batt. While it is relatively easy to set KPIs and rewards for sales staff (eg, sales revenue and bonuses), it is harder to arrange for developers, but Batt believes his approach is well received by employees. “People like the opportunity to put more work in and get more out,” he said.

Brady’s teams are judged on software quality (measured in terms of the number of bugs, memory footprint, recognition rates and so on) as well as timely delivery. They are also kept aware of the company’s margins, costs and revenues. “It certainly makes a difference,” said Brady, as it makes them feel part of the outcome, as well as helping promote innovation and creativity “which I think are squashed in a lot of development houses.”

Individual and group KPIs are not mutually exclusive, so Batt’s approach does not mean there are no individual KPIs. Specific targets are written into individual contracts, along with the rewards for achieving them.

For example, Smartpath uses test-driven development. This cried out for automated testing, so a new staff member was given the job of setting up an appropriate environment. “We now have it, and it’s fantastic,” said Batt.

Other functions implemented this way include internal database management and patch management. One of next year’s projects will be to build a new database administration tool, and the corresponding milestones will be among the KPIs for a staff member.

Rewards need not take the form of bonus payments. VeCommerce sometimes recognises developers who meet KPIs by sending them to major overseas conferences, said marketing manager Martin Riddle.

Testing

On the face of it, testing is a more organised process than design and coding, and therefore more amenable to metrics such as the number of bugs detected per thousand lines of code, but as at Smartpath, Quest’s approach is to increasingly automate the testing process so that QA people become developers of test suites rather than testers.

Schofield leaves testers and his or her manager to take care of day-to-day aspects, but does correlate field reports of modules failing with the staff that tested them, as well as rating the quality of feedback provided by testers to developers. “[Testers] are very much part of the development process,” he said, so they may be involved in discussions about the right approach for the user interface, for instance.

Regular review

Krishnan’s approach is similar to Batt’s in some respects. Very clear job descriptions are important, setting out among other things key result areas (which may be technical or people-oriented) and specific tasks, along with indicators that show whether the desired outcomes have been achieved.

The six-monthly reviews he favours are to check on previous performance and to set quantifiable goals in each result and task area. This should be a collaborative process, one of “aligning mutual interests”, not an opportunity for managers to impose goals on their staff.

This approach means people know what they are supposed to be doing, and it also identifies areas where further skill development is needed. Furthermore, it gives an opportunity to recognise and reward those who have achieved the agreed goals, providing positive reinforcement.

There is a place for qualitative measures in the review process, and Krishnan points out that one objective is to encourage the individual to exceed his or her brief. For example, one key responsibility area is the contribution of good ideas. While it is possible to count the number of ideas floated and accepted, the quality of ideas is more important than their quantity.

Similarly, initiative, leadership and customer service are all aspects that Krishnan wants to encourage and therefore builds into position descriptions, but they are not amenable to quantitative measures.

How do employees react?

“Finding the right people to work in a software development team is very important,” said Batt, and once you’ve got them, incentives work better than a big stick. People then work hard and build quality products, he said.

Krishnan concedes that some people may find his approach too restricting and that it in some cases may contribute to turnover. He suggests a staged implementation can help, beginning with an explanation of the benefits, stressing that it is about performance management rather than performance measurement, and that any skill gaps revealed will be addressed through education. It may be a while before people actually believe this.

In time it becomes a self-management tool, he said, with staff identifying what they have and have not achieved.

Once people are onboard with the idea, more specific metrics and more explicit financial and other rewards for achieving agreed levels of performance can be introduced.

For developers, “it’s not all about the money,” said Brady, so managers need to tap individual motivations.

KPIs should be used to develop individuals, not to punish them for failure, Krishnan insists. “It has to be done in that positive manner, or it fails.”

He believes most of his staff recognises his approach as being mutually beneficial. It did take time to build their trust, but providing positive reinforcement helped.