Temkin Group has released a new research report, “Tech Vendors: Benchmarking Product and Relationship Satisfaction of IT Clients, 2013.” The research firm spoke with 802 IT professionals within North American enterprises with at least $500 million in annual revenue in April 2013.

The survey rates 54 IT vendors across eight criteria: four product areas and four relationship areas. According to the report, Microsoft servers and VMware earned the highest scores for their products while Deloitte Consulting and Trend Micro had the lowest product scores. VMware and SAP Analytics scored highest in the relationship category while Deloitte Consulting and Fujitsu were at the bottom.

This is the second year that Temkin Group has conducted this survey. In 2013 average scores for IT vendors dropped across all eight categories. TechRepublic had a conversation with Managing Partner Bruce Temkin to gain his insights into the survey.

TechRepublic: Average scores in the 2013 report dropped in comparison to 2012, and the range of average scores is 41 to 50 percent. What are your thoughts about what appears to be a high level of dissatisfaction among IT clients?

Bruce Temkin: I think there are two things. I do think there is a relatively low average score across all of those items, which reflects, particularly, the huge opportunity for tech vendors to improve, especially in the relationship areas. A lot of the relationship areas fell well below the product areas. And in terms of the year-over-year change, since this is the second year it’s very hard to extrapolate whether there’s any long term trend going on with that. But we pretty much saw average scores drop across all the eight criteria, the four product and four relationship criteria, which seems to me there is definitely a broad trend heading in the wrong direction.

TechRepublic: Who is the primary market for the report, and how do you think those enterprises will benefit from it?

Bruce Temkin: The report is really focused on two broad markets. One is the vendors–large organizations and their clients. It is helpful for vendors to understand where they are relative to other vendors. And it is valuable for the companies to understand how the different tech vendors rank in these broad categories. The second group is people who care about the B2B customer experience.

This is one of many evaluations we do, like the Temkin Experience Ratings for tech vendors and our NPS benchmark, and all of those are helpful for companies to think in terms of their B2B relationships. For instance, any company that sells a product to a large organization can pretty much think about the same eight criteria that we use, and think about the two broad categories of products and relationships. So there are a number of people who read it for that lens as well.

TechRepublic: What should the executives with the IT vendors surveyed in the report be paying attention to?

Bruce Temkin: Because the data go down year-over-year I think it’s important for a tech vendor to look at where they are relative to their most relevant competitors. We have 54 vendors and they don’t all compete in the same space. So I think it is most important for vendors to look at what your scores are and how you are really doing relative to the two to four vendors that you think you really compete with.

TechRepublic: VMware scored highest in six out of the eight categories. What do you believe is driving its success?

Bruce Temkin: VMware has traditionally focused on the specific needs of their tech buyers. Unlike some of the other tech vendors who have a wider portfolio of products, VMware has a relatively narrow footprint in terms of the technology space that it occupies, which I think helps it be a little more focused on the needs of the customers in that space. And they’ve done a pretty good job of remaining focused on their market.

TechRepublic: Any company with the words “outsourcing” or “consulting” in its name scored lower than 50 percent on the survey. What seems to be the source of their troubles?

Bruce Temkin: I think there are two ways to look at that, even in the context of things we’ve talked about. When you look at the broad market space across these, they’re clearly not seen as positively both in the products and services they offer and the relationships they have compared to the more product-oriented companies.

To me, especially if you look at the bottom of the list, some of those companies that are really low, like Deloitte, who tended to be at the bottom, those companies are really at risk of bad word of mouth and non-renewal rates. Because we have looked at these data and the data are correlated with future purchases, I think you can’t sustain really low performance in your customers’ eyes before it hurts you.

The other more positive view is that the reason these companies are at the bottom of the list is that outsourcing and consulting firms, sort of the flip of VMware, have more complex things. It’s not about doing the exact standard think over and over again where you can create a product or a model and repeat it.

So I think that what they offer, because it’s more complicated, does make it more difficult to consistently deliver on the product and relationship side. But having said that, many of them on the list are doing so bad that there’s no comparison that makes it OK, there’s no way to look at it in a positive way.

TechRepublic: From this report, what is the big take home message for IT clients?

Bruce Temkin: I think that IT clients typically look at the products that are offered to them. IT clients tend to have a reasonably good understanding of how their tech vendors deliver in terms of the products vis-à-vis the other ones they deal with. I think even though those scores are low IT buyers have a sense for where the strengths and weaknesses are.

To me, the big take home message for IT buyers is that they should expect a lot more out of their relationships, and really start looking at the quality of the relationships that they have with those vendors. Whether it’s tech support or the account team or overall cost of ownership, those things vary the most. And I think it’s important for IT buyers to start pushing their vendors to deliver more on that front as well.

TechRepublic: Have there been any attempts at rebuttal by any of the low-scorers on the survey? Or is it too soon?

Bruce Temkin: This is the second year we’ve done it. And we do lots of tech vendor meetings with the NPS score. And we’ve only had one company pass the threshold of a “rebuttal.” And that was last year, so no one’s really pressed us this year. So your question about too soon–who knows. It could catch someone’s eye in three years and they could decide to push back.

But what we try to do and I think we do very well is be very transparent about our methodology and not over-read into the results. And so even in that one case where someone pushed back they didn’t really have anything to push back on other than they didn’t agree with the results, and we’re OK with that. Hopefully we’re incredibly clear on what we did and companies can decide on whether they want to agree with or learn from or ignore the results. That’s fine with us.

TechRepublic: It’s interesting that you mentioned the transparency. I think that everyone can benefit from that.

Bruce Temkin: You know one of the things I’ve learned doing research over the years is that anytime you compare companies with any measure of research, they’re going to be people at the top and people at the bottom. You can be certain that some the people at the bottom are going to be unhappy. But it turns out that even some of the people at the top can be unhappy because they don’t think they are far enough ahead.

So really, you end up with lots of people who are unhappy. So what I’ve learned is that it’s OK for them to be unhappy with the data, but it’s not OK for them to be confused about the approach. So we always try as best we can to explain what we did, so people can make their own judgment about whether the results are meaningful for them or not.

TechRepublic: How can IT job searchers and recent computer science graduates benefit from this survey?

Bruce Temkin: I think they both can learn. I think on the product side, looking at where companies are rated relative to the product dimensions is probably the most important for someone who is in a technical field, who will be going to the development side or the technical side in some of these firms.

If you’re in sales or marketing, you may want to look at the relationship side of these things, of the scores, and identify the leading companies in those areas. If I were looking for a job I would probably want to look at the companies that did better on this survey as opposed to those who did worse.

TechRepublic: What was your inspiration for starting Temkin Group after 12 years at Forrester Research?

Bruce Temkin: I had seen that even in a down economy my research around customer experience was booming. For 14 consecutive quarters I was the most-read Forrester analyst. So I knew there was huge demand for the types of research I was doing and the types of insights we were building. It made sense for me to have more freedom to be able to focus more completely on the space by creating The Temkin Group.

Temkin Group is a customer experience research and consulting firm located in Waban, Mass. The survey, “Tech Vendors: Benchmarking Product and Relationship Satisfaction of IT Clients, 2013” can be purchased and downloaded from the Temkin Group website.