Data Layout Optimization for Petascale File Systems

Download Now Date Added: Nov 2009
Format: PDF

High-Performance Computing (HPC) has crossed the Petaflop mark and is moving forward to reach the Exaflop range. However, while computing resources are making rapid progress, there is a significant gap between processing capacity and data-access performance. In this paper, the authors propose a simple performance model to promote a better integration between the parallel I/O middleware layer and parallel file systems. They show that application-specific data layout optimization can improve overall data access delay considerably for many applications. Implementation results under MPI-IO middleware and PVFS2 file system confirm the correctness and effectiveness of their approach, and demonstrate the potential of data layout optimization in petascale data storage.