Science & Engineering Research Support soCiety (SERSC)
As the access speed gap between DRAM and storage devices such as hard disk drives is ever widening, the I/O module dominantly becomes the system bottleneck. Meanwhile, the map-reduce parallel programming model has been actively studied for the last few years. In this paper, the authors will show empirically show that flash memory based SSD (Solid State Drive) is very beneficial when used as local storage devices in IO-intensive map-reduce applications (e.g. sorting) using Hadoop open source platform. Specifically, they present that external sorting algorithm in Hadoop with SSD can outperform the algorithm run with hard disk by more than 3.