Banking

Begin an audit strategy to monitor vendor performance

Don't wait for a problem to erupt before you begin monitoring how your vendor is performing. Learn ways to begin scheduling audits to take a proactive approach.


Just about any IT manager who has kept watch over an external service provider has a horror story to tell.

“I once brought in a vendor to set up an NT 4.0 server with MS Exchange 5.5,” said Paul Sullivan, a network administrator with GBF Graphics, Inc. “The vendor indicated the project would be accomplished within two days after the local phone provider completed an ISDN circuit. In fact, it took about four weeks.”

Sullivan said that during the installation process, the technician seemed to be constantly coming up with new problems.

”For example, he thought there might be a ‘data storm’ in our hub, which was functioning well for everyone else on it, and our ISP was being ‘uncooperative’ troubleshooting his problems,” he said.

Why don’t you schedule an audit?
Because problems with vendors are often obvious, many busy IT managers, like Sullivan, haven’t found a need to take the time to formally audit the performance of external service providers (ESPs).

“We’ve had a number of them helping with various projects over the years,” Sullivan said, “and a general impression of the process and the end result usage of the product or service is my performance gauge.”

Such informal measurements may be adequate for simple projects, but as the relationships between organizations and ESPs increase in number, scope, and complexity, comprehensive performance audits may provide benefits that justify the time it takes to do them—even when everything seems to be going well.

What is a performance audit?
According to Gartner analyst William Maurer, an ESP audit involves more than just a one-time check for problems. Instead, it’s a continuous improvement process.

“A performance audit is one of the ongoing measurement tools used by both the vendor and the client to see how activities are being performed—to measure performance results against goals,” said Maurer.

He described an example involving a help desk. If somebody has a goal of help desk calls being answered within 30 seconds 95 percent of the time, and that’s not being achieved, then both parties should investigate.

“The service receiver and service provider would analyze why the goal isn’t being met, determine the root cause, and recommend a fix to the problem. That recommendation would then be implemented,” said Maurer.

Scheduling audits
A schedule of audit and performance reviews is one of the most effective mechanisms for ensuring that business value is delivered through external service providers.

“These reviews are typically specified in the contract to ensure that the right level of staffing is devoted to the effort and that the right data is made available to the auditors,” said Maurer.

The review schedule should be synchronized with the deliverables schedule or, on long-term projects, conducted with regular frequency. Quarterly or semiannual audits might make sense for your organization. The review methodology should map directly to the deliverables, such as transactions per month or reduced inventory in hand. But Maurer cautioned that you shouldn’t allow audit schemes to become too complex.

“They must be kept simple so that the report data can be reliably produced without interrupting the business operation of the vendor. Regular performance reviews are typically welcomed by the vendor that wants validation of the value being delivered.”

Developing SLAs
Service-level agreements are tools you can use to help ensure success in managing continuous improvement.

Maurer suggested that developing an SLA is a five-step process:
  • Define the key processes you want to measure.
  • Establish service levels for each activity and compare results to goals.
  • Examine the measured results and determine areas for improvement. Look for root causes of results that don’t conform to expectations.
  • Correct the flawed process.
  • Guide the process to retain the benefits achieved by working with the vendor throughout the relationship.

Among the major reasons an ESP engagement fails is that users don’t address service-level issues adequately during contract negotiations or they fail to require performance-tracking mechanisms after implementation.

“Successful deals are characterized by the requirement for a documented and consistently implemented methodology for service-level management. That methodology should be encapsulated in the SLA,” said Maurer.

Creating a balanced scorecard
Try using a “balanced scorecard” approach. According to Maurer, it provides the most effective criteria for measuring the performance of an ESP. This approach is a methodology that requires monitoring five key performance areas:
  • Productivity
  • Service quality (the degree of effectiveness of application delivery)
  • Achievement of financial targets
  • Client satisfaction
  • Future value (commitments regarding tools, methodology, and training)

“These five areas cover the plans and commitments providers make in order to save you money by being more productive,” Maurer said. “Implementing the balanced scorecard, while rewarding, requires more thought and discipline than historical ESP approaches.”

For ESP relationships to work well, both parties must be willing to come to a common state of expectations and have the discipline to ensure the suppliers follow through on their propositions to save you money.

“Organizations that abdicate management activities to ESPs will pay more and remain reliant on ESPs longer than necessary,” Maurer said. “No matter how comprehensive an ESP contract is, management of the ESP cannot be contracted out.”
What advice would you give an IT manager who is performing an audit on an ESP? Post your comment below or send us a letter.

Editor's Picks

Free Newsletters, In your Inbox