Date Added: Apr 2010
As one enters the multi-core era where contemporary processors integrate multiple processor cores on a single chip and Simultaneous Multi-Threading (SMT), or running multiple hardware threads on a single core has become commonplace, benchmarking consolidated servers that run multiple guest VMs on the same physical hardware continues to remain a matter of discussion. A benchmarking methodology for consolidated servers should quantify both total system throughput and per-VM performance in order to provide a meaningful and precise performance characterization. The current benchmarking methodologies focus on total system throughput (i.e., across all VMs in the system), and do not take into account per-VM performance. This paper presents two performance metrics - Total Normalized Throughput (TNT) to characterize total system performance; and Average Normalized Reduced Throughput (ANRT) to characterize per-VM performance, thereby contradicting VMmark that says one system performs better than another. TNT and ANRT show that both platforms represent different trade-offs in total system throughput versus per-VM performance. Ideally, it is argued that a meaningful and precise server consolidation performance characterization should use both metrics. Taking into account the various existing benchmarking frameworks, the paper compares them with the TNT and ANRT metrics thereby attempting to help reach a consensus on how to benchmark consolidated servers. It is believed that a benchmarking methodology that scores both overall system performance and per-VM performance metrics are key in order to do a meaningful and precise performance characterization.