The Unified Logging Infrastructure for Data Analytics at Twitter
In recent years, there has been a substantial amount of work on large-scale data analytics using Hadoop-based platforms running on large clusters of commodity machines. A less-explored topic is how those data, dominated by application logs, are collected and structured to begin with. In this paper, the authors present Twitter’s production logging infrastructure and its evolution from application-specific logging to a unified “Client events” log format, where messages are captured in common, well-formatted, flexible Thrift messages. Since most analytics tasks consider the user session as the basic unit of analysis, they pre-materialize “Session sequences”, which are compact summaries that can answer a large class of common queries quickly.