Opening the Black Boxes in Data Flow Optimization

Many systems for big data analytics employ a data flow abstraction to define parallel data processing tasks. In this setting, custom operations expressed as user-defined functions are very common. The authors address the problem of performing data flow optimization at this level of abstraction, where the semantics of operators are not known. Traditionally, query optimization is applied to queries with known algebraic semantics. In this paper, they find that a handful of properties, rather than a full algebraic specification, suffice to establish reordering conditions for data processing operators. They show that these properties can be accurately estimated for black box operators by statically analyzing the general-purpose code of their user-defined functions.

Provided by: Technische Universität Berlin Topic: Data Management Date Added: May 2012 Format: PDF

Find By Topic