Toward Massive Query Optimization in Large-Scale Distributed Stream Systems

Existing distributed stream systems adopt a tightly-coupled communication paradigm and focus on fine-tuning of operator placements to achieve communication efficiency. This kind of approach is hard to scale (both to the nodes in the network and the users). In this paper, the authors propose a fundamentally different approach and present the design of a middleware for optimizing massive queries. Their approach takes the advantages of existing Publish/Subscribe systems (Pub/Sub) to achieve loosely-coupled communication and to "Intelligently" exploit the sharing of communication among different queries.