Provided by: Johns Hopkins University
Date Added: Nov 2012
Stream processing methods and online algorithms are increasingly appealing in the scientific and large-scale data management communities due to increasing ingestion rates of scientific instruments, the ability to produce and inspect results interactively, and the simplicity and efficiency of sequential storage access over enormous datasets. This paper will showcase the authors' experiences in using off-the-shelf streaming technology to implement incremental and parallel spectral analysis of galaxies from the Sloan Digital Sky Survey (SDSS) to detect a wide variety of galaxy features. The technical focus of the paper is on a robust, highly scalable Principal Components Analysis (PCA) algorithm and its use of coordination primitives to realize consistency as part of parallel execution.