Work Queue + Python: A Framework for Scalable Scientific Ensemble Applications
Even with the increase in the number and variety of computer resources available to research scientists today, it is still challenging to construct scalable distributed applications. To address this issue, the authors developed Work Queue, a flexible master/- worker framework for building large scale scientific ensemble applications that span many machines including clusters, grids, and clouds. In this paper, they describe Work Queue and then present the Python-Work-Queue module, which enables scientists to take advantage of their Work Queue framework while using the Python programming language. To demonstrate the module's flexibility and power, they examine two distributed scientific applications, RepExWQ and Folding@work.