Data Centers

In-Situ I/O Processing: A Case for Location Flexibility

Free registration required

Executive Summary

Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process output data during simulation time, "in-situ", and before placing data on disks. This paper argues for flexibility in the implementation of such in-situ data analytics, using measurements and a performance model that demonstrate the potential advantages and limitations of performing analytics at different levels of the I/O hierarchy, including on a machine's compute nodes vs. on separate "staging" nodes dedicated to analysis tasks. Model and measurement results are guided by realistic large-scale applications running on leadership class machines, and I/O and analytics actions are described as computational dataflow graphs - termed I/O graphs - that combine data movement with 'in transit' operations on data as it is being moved across the I/O hierarchy.

  • Format: PDF
  • Size: 393.94 KB