Data Centers

Benchmarking: Tools of the trade

Are you in need of benchmarking tools? In this Daily Drill Down, James McPherson lists many of the best benchmarking application suites out there.


In a recent Daily Drill Down, “Benchmarking basics,” I gave an overview of benchmarking. For those of you unfamiliar with benchmarking tools, this Daily Drill Down covers some of the major test suites you can procure to perform your own benchmarking comparisons. I’ve organized this Daily Drill Down by applications available for complete system tests and for subsystem tests.

I didn’t cover software testing, as there aren’t many software-testing suites. Most software tests must be custom developed to stress various aspects of each application and generate useful results. It is especially difficult to compare similar programs from different vendors, as a “fair” test can be difficult to identify.

Complete system tests
Each of the following applications is intended for testing complete systems. Each benchmark package has its own strengths and weaknesses, so be sure to choose a package that is relevant to your situation.

Complete system test suites can be used to test processors, motherboards, memory, and hard drives in real world situations, as long as all other factors are kept constant. Remember that driver optimizations will play a large part in overall system performance, so the results for a particular component may not reflect the results on another operating system.

Business Winstone 2001
Publisher: ZDNet
Platform: Windows 9x, NT, 2000
Cost: $5 shipping & handling cost for CD. Download is not available.
Web site: www.winstone.com/

This is an overall system evaluation package that runs actual 32-bit applications through various scripted activities and times their performance. Business Winstone 2001 is a good general-purpose testing suite as it provides a relatively wide variety of test conditions.

The suite is broken down into four categories running the following applications:
  • Business: Corel WordPerfect 8, Lotus SmartSuite 97, Microsoft Office 98
  • Dual Processor: MicroStation 95, Photoshop 4, Visual C++ 5.0
  • High End: AVSExpress 3.4, FrontPage 98, MicroStation 95, Photoshop 4.0, Premiere 4.2, Sound Forge 4.0, Visual C++ 5.0

To further minimize error, the application can, and should, be configured to perform an initial “training” run followed by five monitored runs separated by rebooting and defragmenting the drives. The training run eliminates the many small events (creating cache files, working directories, verifying registry keys) that occur behind the scenes the first time a program is run. This is especially important for certain self-optimizing systems like the Transmeta Crusoe that can actually improve their performance by learning the application. By choosing the best of five tests, you eliminate much of the random error that occurs in such complex tests on current hardware.

Because the Winstone suites include significant portions of the applications involved, and because ZDNet doesn’t have permission to post these applications on the Web, Winstone products are only available by ordering a CD.

Winstone Content Creation 2001
Publisher: ZDNet
Platform: Windows 9x, NT, 2000
Cost: $5 shipping & handling cost for CD
Web site: www.winstone.com/

Content Creation is perhaps the single best test suite for determining real-world system performance. Like Business Winstone, Content Creation uses actual 32-bit applications to evaluate a system’s capabilities. Unlike its sibling, however, Content Creation involves significant multitasking. A number of applications are left running and the test scripts switch between them, just as a typical user would.

The test applications are typical multimedia applications that would be used in concert to stress a machine. The included applications are Adobe Photoshop 5.5, Adobe Premiere 5.1, Macromedia Director 8.0, Macromedia Dreamweaver 3.0, Netscape Navigator 4.73, and Sonic Foundry Sound Forge 4.5.

The results are a unitless value given relative to the base system performance (Dell Dimension XPS P5-233), which provides a score of 10. A result of 20 is twice as fast as the base machine; a 50 is five times as fast.

Like Business Winstone, Content Creation is only available on CD.

WebMark2001
Publisher: Business Applications Performance Corporation (BAPCo) and MadOnion.com
Platform: Windows NT/2000 (server), Windows 98 SE/2000 (client)
Cost: CD-ROM $299.95 plus $14.99 for shipping and handling
Web site: labpro.madonion.com/

WebMark2001 is a client testing utility that attempts to identify the performance of systems in an Internet client/server situation. WebMark2001 requires significantly more effort than other benchmark tools to configure the IIS server along with the client. It uses seven Web site configurations in three categories:
  • Business-to-business
  • Business-to-consumer
  • Business/corporate intranet

The tests attempt to model the most common types of Internet client/server scenarios. It is a thorough test of most Web functions, including Adobe Acrobat, QuickTime, Java/script, Flash, SSL, XML, RealAudio/Video, and Microsoft Media.

WebMark tests are useful for testing thin clients and full PCs intended for Web terminals or network segments when a large number of clients are run against the server.

Sysmark2000
Publisher: Business Applications Performance Corporation (BAPCo) and MadOnion.com
Platform: Windows 2000, NT, 98, 95
Cost: CD-ROM $59.99 plus $4.99 for shipping and handling
Web site: labpro.madonion.com/

Sysmark2000 consists of two high-load application suites: Office Productivity and Internet Content Creation. The suite runs each application individually, but under sufficiently grueling conditions that multitasking them is not necessary.

Office Productivity suite consists of CorelDraw 9.0, Paradox 9.0, MS Word 2000, Excel 2000, PowerPoint 2000, Dragon NaturallySpeaking Preferred 4.0, and Netscape Communicator 4.61. The Internet Content Creation section is composed of Adobe Premiere 5.1, Adobe Photoshop 5.5, Avid Elastic Reality 3.1, Corel’s Bryce 4, and Microsoft Windows Media Encoder 4.0.

Sysmark2000 is appropriate to test workstation-class systems intended for intensive business computing or graphics manipulation. The inclusion of multiple applications of the same type improves the likelihood that this test will be relevant to your IT situation.

Games
Publisher: various
Platform: all
Cost: varies

As many of you know, games make excellent applications for testing system performance. The current crop of games, especially first person shooters like Quake III and Unreal Tournament will readily stress the interaction of processor, memory, and graphics systems.

Component tests
Each of the following applications attempts to determine the performance of components or subsystems. The tests vary dramatically in their focus. Some, like Stream, use customized routines that evaluate the maximum potential performance of a component, which can have very little relationship to real-world performance of standard applications but can be of immense use to programmers and developers. Other tests use recordings of actual program operations to provide results more immediately useful to IT staffers and purchasing agents. The type of test should be tailored to the specific situation to avoid generating irrelevant data.

WinBench 99 v1.2
Publisher: ZDNet
Platform: Windows 9x, NT, 2000
Cost: $5 shipping & handling cost for CD or free download
Web site: www.winbench.com/

WinBench is similar to the Winstone tests in that it uses actual 32-bit applications in its evaluation of various subsystems. WinBench has the recorded action of those applications, however, not the applications themselves. This helps isolate the component in question.

This suite provides the following information:
  • Disk: Transfer speed (MB/s), access time (ms), CPU utilization (percentage)
  • Graphics: WinMark (unitless time test), performance (megapixel/s)
  • Processor: CPUmark and FPUmark (unitless time test), utilization (percentage)
  • Audio/Video: dropped frames, audio breaks, nominal speed (percentage), maximum frame rate (frames/s), CPU utilization (percentage), performance (megapixels/s)

Sandra 2001
Publisher: SiSoftware
Platform: Windows 9x, NT, 2000
Cost: free downloadable demo version, $29 to download full version, $29 plus shipping and handling for disks.
Web site: www.sisoftware.demon.co.uk/

Sandra 2001 is a general hardware-analysis tool that can be used to catalog a system. It identifies a variety of system information from fonts, DLLs, and drivers to OLE information. It also benchmarks CPUs, multiprocessor performance, hard drive and CD-ROM/DVD drives, memory bandwidth, and network performance.

Sandra excels at analyzing the memory and CPU systems. The tests are very specialized and are more suited for determining the potential of a system than the real-world performance. Combined with Sandra’s system cataloging ability and low price, however, Sandra is an excellent tool for most IT shops.

STREAM
Publisher: University of Virginia
Platform: DOS, Windows 9x, NT, 2000, Linux, Macintosh
Cost: free download
Web site: www.cs.virginia.edu/stream/ref.html

This is a dedicated memory subsystem test developed by the University of Virginia’s C.I.S. program. It uses special vector math routines that exceed processor cache sizes to monopolize the main system memory. This can find bottlenecks in memory subsystems.

STREAM is capable of determining the potential performance of various types of memory and is one of the key tools used to compare Rambus, SDRAM, and DDR memory and memory controllers. This test is very specialized and is only appropriate for IT staff capable of understanding the ramifications and implications of the results.

Conclusion
Benchmarking a system properly calls for special sets of utilities. In this Daily Drill Down, I’ve presented a list of applications that can help you test whole systems and components.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

Editor's Picks