Institute of Electrical & Electronic Engineers
Today, most tools for processing big data are batch-oriented. However, many scenarios require continuous, online processing of data streams and events. The authors present ESC, a new stream computing engine. It is designed for computations with real-time demands, such as online data mining. It offers a simple programming model in which programs are specified by Directed Acyclic Graphs (DAGs). The DAG defines the data flow of a program; vertices represent operations applied to the data. The data which are streaming through the graph are expressed as key/value pairs. ESC allows programmers to focus on the problem at hand and deals with distribution and fault tolerance. Furthermore, it is able to adapt to changing computational demands.