Relative Factors in Performance Analysis of Java Virtual Machines

Many new Java runtime optimizations report relatively small, single-digit performance improvements. On modern virtual and actual hardware, however, the performance impact of an optimization can be influenced by a variety of factors in the underlying systems. Using a case study of a new garbage collection optimization in two different Java virtual machines, the authors show the relative effects of issues that must be taken into consideration when claiming an improvement. They examine the specific and overall performance changes due to the optimization and show how unintended side-effects can contribute to, and distort the final assessment. The experience shows that VM and hardware concerns can generate variances of up to 9.5% in whole program execution time.