Data Centers

Intel trots out misleading benchmark charts

By · March 21, 2007, 6:53 AM PST

Image 1 of 7

  • Intel's energy efficiency slide is like a comparison of apples to eggplant.

    In a manipulative attempt to demonstrate leadership over AMD for the press and analysts, Intel twists benchmark data to make it look as though it's latest Xeons are outperforming the best AMD has to offer. Yet, upon closer inspection, most of the data being shown is irrelevant.

    Using line charts to present benchmark data is tricky to begin with. Usually, you have performance points going up the Y axis and along the X axis, you might have variable loads. But you'd never do what Intel has done here.

    In this graph designed to show leadership in energy efficiency, not only do the benchmarks used change from one data point to the next, but so too do the chips being compared (as evidenced by the extensive footnoting). Whereas the Intel chip stays constant (a dual core Xeon 5160), the AMD chip to which it's compared changes from one benchmark to th next. In some cases, it's the newest chip. In others, an older generation Opeteron is used. Still others, Intel is comparing its dual core to an AMD single-core

    To compound matters, for two of the benchmarks that show Intel taking a commanding lead, both of the benchmarks are six year old benchmarks for which newer versions exit. In fact, of the last ten benchmarks to the right side, all but one compare the Xeon 5160 to a chip it shouldn't be compared to, or use a retired benchmark.

    Where the little yellow question mark appears, we don't even know what AMD chip was used there. It doesn't say. Maybe it was an old 32-bit Athlon?

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

  • Single Core chips broken out.

    This slide, taken from a presentation give one week after the previous slide appeared, is a slight improvement in that Intel is calling out single vs. dual core comparisons. But even so, the rest of the problems remain.

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

  • More Apples vs. Eggplants, this time around performance

    Although we don't have all the annotations on this slide like we did on the first, we could have simply taken them from the first slide and overlayed them onto this one. Pretty much every benchmarking felony committed on the first slide in this presentation was committed on this slide as well, thus completely calling into question Intel's characterization of its performance as "Breakaway." Based on this data, it's nothing of the sort. Would Intel consider the Opteron's advancements over its old 386 technology a sign of a breakaway?

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

  • More benchmarking silliness

    In one version of this slide, Intel cites two SAP benchmarks in declaring "leadership benchmarks" of it's latest greatest Xeons over AMD. But the footnotes reveal that the SAP performance differences are over AMD's older family of Opterons (the 885). In fact, they're the 2.6Ghz 885's which weren't even the fastest versions of those chips. AMD had a 2.8Ghz version as well.

    As if this slide doesn't have enough problems, it too cites the 6 year old SPEC benchmarks for which newer versions have bene issued by their authors.

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

  • Oops, missing footnote

    In this version of the previous slide (the version used to show the press and analysts on February 28th), the fine print that shows how the SAP 4P benchmark was not run against the latest greatest Opteron (the 2200SE) is missing. With that footnote missing, it would be easy for the press to assume it was against the newest generation of Opteron processors. But in reality, as can be seen fro the previous slide, it was not.

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.I

  • Why bother with footnotes?

    We were shocked when we saw this benchmark chart. There were so many footnotes to denote how the chip being compared to was not an Opteron 2XXX (as the headings say) that we wondered why not do it the other way around? Instead of saying which chips were not an Opteron 2XXX, footnote which ones were. There would have been fewer footnotes!
    Check out the top right hand quadrant of this chart. The heading says "vs. Opt 2XXX." Yet, not one single system under that was an Opteron 2XXX. We were imagining the press and analysts looking at this slide, not paying attention to any of the footnotes to realize that there wasn't a single shred of truth t the heading.

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

  • Even worse than the last slide

    In this slide, Intel even had the audacity to use big bold text that says "Continued Leadership vs. Latest Opteron." Yet, 21 of 32 results shown don't involve the latest Opteron or even a member of th the latest Opteron family. And again, just like with the last slide, the top right hand quadrant of data has "vs. Opt 2XXX" in the heading. But none of the chips listed below it were an Opteron 2XXX. Everyone of them had to be footnoted.

    For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

Intel's energy efficiency slide is like a comparison of apples to eggplant.

In a manipulative attempt to demonstrate leadership over AMD for the press and analysts, Intel twists benchmark data to make it look as though it's latest Xeons are outperforming the best AMD has to offer. Yet, upon closer inspection, most of the data being shown is irrelevant.

Using line charts to present benchmark data is tricky to begin with. Usually, you have performance points going up the Y axis and along the X axis, you might have variable loads. But you'd never do what Intel has done here.

In this graph designed to show leadership in energy efficiency, not only do the benchmarks used change from one data point to the next, but so too do the chips being compared (as evidenced by the extensive footnoting). Whereas the Intel chip stays constant (a dual core Xeon 5160), the AMD chip to which it's compared changes from one benchmark to th next. In some cases, it's the newest chip. In others, an older generation Opeteron is used. Still others, Intel is comparing its dual core to an AMD single-core

To compound matters, for two of the benchmarks that show Intel taking a commanding lead, both of the benchmarks are six year old benchmarks for which newer versions exit. In fact, of the last ten benchmarks to the right side, all but one compare the Xeon 5160 to a chip it shouldn't be compared to, or use a retired benchmark.

Where the little yellow question mark appears, we don't even know what AMD chip was used there. It doesn't say. Maybe it was an old 32-bit Athlon?

For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.
Related Topics:
Data Centers Hardware Virtualization Networking Storage Cloud
Show Comments