Timely Offloading of Result-Data in HPC Centers

High performance computing is facing an exponential growth in job output dataset sizes. This implies a significant commitment of supercomputing center resource - most notably, precious scratch space - in handling data staging and offloading. However, the scratch area is typically managed using simple "Purge policies", without sophisticated "End-user data services" that are required to balance center's resource consumption and user serviceability. End-user data services such as offloading are performed using point-to-point transfers that are unable to reconcile center's purge and users delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant.