Distributed Monitoring of Conditional Entropy for Anomaly Detection in Streams

In this paper the authors consider the problem of monitoring information streams for anomalies in a scalable and efficient manner. They study the problem in the context of network streams where the problem has received significant attention. Monitoring the empirical Shannon entropy of a feature in a network packet stream has previously been shown to be useful in detecting anomalies in the network traffic. Entropy is an information-theoretic statistic that measures the variability of the feature under consideration. Anomalous activity in network traffic can be captured by detecting changes in this variability.