Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem

With a goal of supporting the timely and cost-effective analysis of Terabyte datasets on commodity components, the authors present and evaluate StoreTorrent, a simple distributed filesystem with integrated fault tolerance for efficient handling of small data records. The contributions include an application-OS pipelining technique and metadata structure to increase small write and read performance by a factor of 1-10, and the use of peer-to-peer communication of replica-location indexes to avoid transferring data during parallel analysis even in a degraded state.