Every parcel of digital information—including the email you send, Skype calls you make, and websites you load—is transmitted across the web in a formatted piece of structured data known as a "packet." Inside this packet is structured metadata that assures your data is routed to the proper destination. Analyzing these packets is a process known as deep packet inspection (DPI), and the practice is employed daily by enterprise companies, internet service providers (ISPs), and media companies.
TechRepublic's smart person's guide is a routinely updated "living" precis loaded with up-to-date information about how deep packet inspection works, who it affects, and why it's important.
- What is deep packet inspection? A network packet is a formatted and discreet unit of data. Deep packet inspection is a method of analysis that dissects network data to extract useful metadata.
- Why deep packet inspection matters: Deep packet inspection illuminates network trends, helps ISPs optimize bandwidth and throughput, and can reveal user behavior.
- Who deep packet inspection affects: Because deep packet inspection inherently involves exposing sensitive data, IT departments at enterprise companies, ISPs, and consumers are most impacted by the practice.
- When deep packet inspection is happening: Deep packet inspection has been a useful IT tool for nearly two decades. As the internet evolves to include mobile and IoT devices, deep packet inspection is being used more and more frequently.
- How to get started with deep packet inspection: First, learn more about the process. Next, search GitHub and other code repositories for open source tools. Finally, speak with your IT department to learn more about how your company performs deep packet inspection on a daily basis.
SEE: Cybersecurity in 2017: A roundup of predictions (Tech Pro Research)
What is deep packet inspection?
Internet traffic is composed of small bundles of data known as packets. Packets wrap digital information in a cocoon of metadata that identifies traffic source, destination, content, and other pieces of valuable details. Analyzing digital traffic is a lot like analyzing automobile traffic: Patterns reveal useful insights. By studying metadata like headers using deep packet inspection (DPI) network specialists can learn how best to optimize servers to reduce overhead, detect and deter hackers, combat malware, and glean intimate details about user behavior.
Although DPI has a number of uses, the practice is rooted in enterprise network security. Sniffing traffic in and out of a network is understandably useful for preventing and detecting intrusions. Detecting and blocking the IP of malicious traffic is particularly effective at fending off buffer overflow and DDoS attacks.
DPI is also used by internet service providers. If packets are mail, ISPs are the postal service and have access to unencrypted web traffic as well as packet metadata like headers. This provides ISPs with an abundance of useful information, and the companies leverage access to user data in a number of ways. Most ISPs in the United States are allowed to turn user data over to law enforcement agencies. Additionally, many ISPs use consumer data to target advertising, analyze file sharing habits, and tier access service and speeds.
- Deep Packet Inspection: What you need to know (TechRepublic)
- Why Deep Packet Inspection still matters (TechRepublic)
- A Review on Protocol Reverse Engineering (TechRepublic)
- ISPs keep their distance from deep packet inspection (ZDNet)
- How to use VPN to defeat deep packet inspection (CNET)
SEE: Three ways encryption can safeguard your cloud files (Tech Pro Research)
Why deep packet inspection matters
Although packet-sniffing is an archaic tactic, due to the sheer scale of connected devices DPI is more relevant today than in prior epochs. DPI is germane for three primary reasons:
- The scale of connectivity. The internet today, particularly mobile, is more important now to more people for more reasons than ever before. Every company and organization relies on network inspection technology to optimize traffic, reduce overhead, and fend off cyberattacks. DPI isn't the only line of defense, but for many organizations, scanning and analyzing packets is the first line of defense.
- The burgeoning IoT market. Like the mobile market before it, IoT inherently means millions of additional devices will come online in the next few years. Many contemporary IoT devices often lack standard firmware and security standards that could protect the devices from being lassoed into a zombie botnet. DPI will shield ISPs and networks from IoT DDoS attacks and help security analysts learn more about critical IoT security flaws.
- Privacy concerns are mainstream. DPI helps media companies learn about customers in ways unimaginable 10 years ago. Every page you load and every piece of communication you send is filtered and routed through an ISP. No longer "dumb pipes," internet service providers are vertically integrating with media companies (the Comcast NBCUniversal and AT&T Time Warner mergers are two examples) and leveraging their data to target consumers with advertising and assist law enforcement agencies with intelligence gathering.
- BlindBox: Deep Packet Inspection over Encrypted Traffic (TechRepublic)
- Next-generation firewalls: Security without compromising performance (TechRepublic)
- Aerohive's new IoT security solution could have blocked Dyn DDoS attacks, company claims (TechRepublic)
- Berners-Lee: Deep packet inspection compromises Net integrity (ZDNet)
- Web monitoring for ads? It may be illegal (CNET)
SEE: Interview with a hacker: S1ege from Ghost Squad Hackers (TechRepublic)
Who deep packet inspection affects
Beyond enterprise and SMB companies, DPI is used primarily by:
- Media companies. Media companies have a storied history of consolidation. When ISPs buy media companies they combine broadcast data with digital data to determine everything from television and web programming to corporate and consumer internet service prices.
- Law enforcement agencies. It is legal, and sometimes required, that ISPs gather and share DPI-gathered data for crimes involving intellectual property violation and drug and human trafficking.
- Consumers. Most consumers are aware that, love it or lump it, personal data is for sale. Most consumers are likely unaware that their ISP is probably analyzing, anonymizing, and reselling personal browsing data to advertising companies.
- European carriers threaten net neutrality with ad blocking (TechRepublic)
- Maxthon browser is a wolf in sheep's clothing (TechRepublic)
- IoT projects: Allow for failure but plan for success (Tech Pro Research)
- Report: Improper use of Deep Packet Inspection could be Internet game-changer (ZDNet)
- IP blocking, privacy-busting packet inspection (CNET)
When deep packet inspection is happening
Deep packet inspection, known also as full packet inspection or data packet inspection, dates back to the ARPAnet. The ARPAnet predated today's internet and was the first computer network to use the TCP/IP data transfer protocol. Managing proto-packets helped engineers learn how to use header and metadata information to mitigate security challenges related to UNIX.
ARPAnet went dark in 1990, but as the modern internet went mainstream so too did TCP/IP challenges. A model called Open Systems Interconnect (OSI) was developed by network engineers in the 1980s to standardize metadata congealed by the mid-1990s. By formalizing packet metadata tiers, OSI allowed for a variety of statistical analysis. For example, secondary headers, known as stateful or shallow data, allow information to be properly routed but cut bandwidth.
Tiered packet metadata also meant ISPs could more easily discriminate against types of data. With the boom of Web 2.0 and mobile in the early 2000s, ISPs realized deep packets could inspire new business models. Net neutrality has been a hot topic for nearly two decades, and deep packet inspection technology has transformed pipe owners in to data owners.
- Breaking the Pattern: The Importance of Deep Packet Inspection for Intrusion Prevention (TechRepublic)
- NethServer 7: Major improvements make choosing this server a no-brainer for SMBs (TechRepublic)
- IT Communication Plan: Raise security awareness with regular emails (Tech Pro Research)
- Ask a hacker: Top four anti-surveillance apps (ZDNet)
- Trend Micro Deep Security Deep Packet Inspection & Firewall (CNET)
SEE: Five ways small companies can get ahead through technology sharing (Tech Pro Research)
How to get started with deep packet inspection
Step 1: Read and learn. DPI is a time-tested tactic, but the IT industry is innovating quickly. The hottest trend in DPI is dedicated hardware. Cisco and several other networking manufacturers have created routers that specialize in packet sniffing and network awareness.
Step 2: A number of free, open source tools exist to help customize DPI for your organizational needs. GitHub hosts a number of free deep packet inspection tools. nDPI is an open source tool that supports a laundry list of customizable plugins.
Step 3: Speak with your IT legal department. DPI is a fundamental component of IT, and experienced network managers should be able to familiarize you with how your company gathers data. If you work in IT, make sure to speak with your legal or Standards and Practices department for usage guidance. Though it's often legal to gather customer data, the industry is heavily regulated. DPI can be a powerful tool. Never break the law, violate company policy, or behave unethically.
- Solution Guide - Advanced Protection Beyond the Firewall (TechRepublic)
- Solving the mystery of next-generation firewalls (TechRepublic)
- Enterprise encryption: Trends, strategic needs, and best practices (Tech Pro Research)
- U.K. spy agencies plan to install Web snooping 'black boxes' (ZDNet)
- Verizon draws fire for monitoring app usage, browsing habits (CNET)
- Free Nethserver might be the small business server you're looking for (TechRepublic)
- 10 books every small business owner and entrepreneur needs to read (TechRepublic)
- How to test drive the free OnlyOffice Community Edition server (TechRepublic)
- How to install a LAMP stack on CentOS (TechRepublic)
- How to back up over a network using rsync (TechRepublic)
- How to install CentOS Web Panel for easy server administration (TechRepublic)
- How to install NGINX on your CentOS server (TechRepublic)
- Poll: What new cybersecurity trends will dominate 2017? (TechRepublic)
- 2017 cybercrime trends: Expect a fresh wave of ransomware and IoT hacks (TechRepublic)
- Gallery: The 10 biggest business hacks of 2016 (TechRepublic)
- Interview with a hacker: Gh0s7, leader of Shad0wS3c (TechRepublic)
- Five essential cybersecurity audiobooks (TechRepublic)
- Five essential cybersecurity podcasts for IT professionals (TechRepublic)
- Cyberwar: The smart person's guide (TechRepublic)
- IT Security for the Mobile Workforce (Tech Pro Research)
- HPE bolsters hybrid IT lineup for SMBs (ZDNet)
- When it comes to IT budgets, bigger isn't always better (ZDNet)
Dan Patterson has nothing to disclose. He does not hold investments in the technology companies he covers.
Dan is a Senior Writer for TechRepublic. He covers cybersecurity and the intersection of technology, politics and government.