Efficient Access to Many Small Files in a Filesystem for Grid Computing

Many potential users of grid computing systems have a need to manage large numbers of small files. However, computing and storage grids are generally optimized for the management of large files. As a result, users with small files achieve performance several orders of magnitude worse than possible. Archival tools and custom storage structures can be used to improve small-file performance, but this requires the end user to change the behavior of the application, which is not always practical. To address this problem, the authors augment the protocol of the Chirp filesystem for grid computing to improve small file performance.