Inspector Gadget: A Framework for Custom Monitoring and Debugging of Distributed Dataflows
The authors consider how to monitor and debug query processing dataflows, in distributed environments such as Pig/Hadoop. Their work is motivated by a series of informal user interviews, which revealed that monitoring and debugging needs are both pressing and diverse. In response to these interviews, they created a framework for custom dataflow instrumentation, called Inspector Gadget (IG). IG makes it easy to write a wide variety of monitoring and debugging behaviors, and attaches seamlessly to an existing, unmodified dataflow environment such as Pig.