Benchmarking File System Benchmarking: It is Rocket Science
The quality of file system benchmarking has not improved in over a decade of intense research spanning hundreds of publications. Researchers repeatedly use a wide range of poorly designed benchmarks, and in most cases, develop their own ad-hoc benchmarks. The authors' community lacks a definition of what they want to benchmark in a file system. They propose several dimensions of file system benchmarking and review the wide range of tools and techniques in widespread use. They experimentally show that even the simplest of benchmarks can be fragile, producing performance results spanning orders of magnitude.