GridTorrent Framework: A High-Performance Data Transfer and Data Sharing Framework for Scientific Computing
Source: Indiana University
Large amount of data that is often stored in many thousands of files is created as part of today's geographically distributed scientific computation and collaboration environments. Managing and transferring large volumes of data sets present a significant challenge and are often a bottleneck in the scientific computing community. This paper introduces an architecture to manage data distributions in a collaborative fashion through a GridTorrent Framework (GTF) whose data transfer mechanism inspired by Bittorrent. This paper presents performance experiment data that compares the framework to Parallel TCP (PTCP) and Bittorrent. Experimental results conducted suggest that using GridTorrent for large data set has significant advantages over parallel TCP in LAN and WAN type of computer networks.