A Case for Scaling HPC Metadata Performance Through De-Specialization
Lack of a highly scalable and parallel metadata service is the Achilles heel for many cluster file system deployments in both the HPC world and the Internet services world. This is because most cluster file systems have focused on scaling the data path, i.e. providing high bandwidth parallel I/O to files that are gigabytes in size. But with proliferation of massively parallel applications that produce metadata-intensive workloads, such as large number of simultaneous file creates and large-scale storage management, cluster file systems also need to scale metadata performance.