Massachusetts Institute of Technology
The archiving of Internet traffic is an essential function for retrospective network event analysis and forensic computer communication. The state-of-the-art approach for network monitoring and analysis involves storage and analysis of network flow statistic. However, this approach loses much valuable information within the Internet traffic. With the advancement of commodity hardware, in particular the volume of storage devices and the speed of interconnect technologies used in network adapter cards and multi-core processors, it is now possible to capture 10Gbps and beyond real-time network traffic using a commodity computer, such as n2disk. Also with the advancement of distributed file system (such as Hadoop, ZFS etc.) and open cloud computing platform (such as OpenStack, CloudStack and Eucalyptus etc.), it is practical to store such large volume of traffic data and fully in-depth analyses the inside communication within an acceptable latency.