Distributed, Application-Level Monitoring for Heterogeneous Clouds Using Stream Processing
As utility computing is widely deployed, organizations and researchers are turning to the next generation of cloud systems: federating public clouds, integrating private and public clouds, and merging resources at all levels (IaaS, PaaS, SaaS). Adaptive systems can help address the challenge of managing this heterogeneous collection of resources. While services and libraries exist for basic management tasks that enable implementing decisions made by the manager, monitoring is an open challenge. The authors define a set of requirements for aggregating monitoring data from a heterogeneous collections of resources, sufficient to support adaptive systems. They present and implement an architecture using stream processing to provide near-real-time, cross-boundary, distributed, scalable, fault-tolerant monitoring.