Linux

Troubleshoot network problems with tcpdump on Linux

See how to use tcpdump to listen to and record traffic on a network segment and then analyze it.

Tcpdump is a network utility that listens to and records traffic on a network segment. This can be highly useful in troubleshooting and monitoring network activity. When preliminary troubleshooting does not solve a network problem, sometimes it is only at the packet or frame level that you will find your answer. That's where tcpdump comes into play.

When working with tcpdump, you can specify a large number of variables to help pinpoint the problem you're trying to diagnose. This article will show you how to best use tcpdump on Linux for network troubleshooting.

Getting tcpdump
Tcpdump, which is distributed under a BSD-style license, should be fairly easy to obtain. You can download the source or find CVS information on the maintainer’s site, tcpdump.org, or from various Linux software repositories around the Web. Compiling from source does not differ greatly from most Linux software. A simple ./configure, make, and make install should get you up and running. You should also have no problems finding distribution-specific releases.

Tcpdump for Windows
A Windows version of tcpdump, called WinDump, is now available. It is essentially the same as tcpdump for Linux/UNIX, and the instructions in this article should translate to WinDump.

Running tcpdump
Tcpdump needs to be run as root under Linux in order to be able to sniff network packets. This is a security feature, as you certainly wouldn’t want to enable users to capture clear text passwords and private e-mails (both of which are possible using tcpdump). The network interface you want to use to listen to network traffic needs to be placed into promiscuous mode, which basically means it will listen to all traffic on the local segment, even traffic that is directed toward other systems.

Before running tcpdump, it's good to decide exactly what you are looking for. Even small networks are extremely chatty, and it will help if you can narrow the focus of your search. Knowledge of standard ports and protocols will be useful. For instance, if you're debugging simple Telnet activity, it's important to know that it operates on TCP port 23. Pinpointing exactly what you need to look for is half the battle. Usually if you reach this stage in the troubleshooting process, you will already have some idea of what you are looking for.

For example, I was recently confronted with an issue involving Macintosh computers and DNS resolution. Basically, the Macs were failing lookups to some servers and not to others. The particular servers that failed sat behind a BSD box acting as a load balancer. When tcpdump was used to snoop packets from specific Mac test boxes, it became apparent that the requests were hitting the server. But due to the SNAT in place on the BSD box, the responding source port was being modified. The Macs were failing to acknowledge the requests because they were waiting for an answer on the same outgoing port. Tcpdump was able to capture the packets and provide us with the answer we needed.

Using tcpdump to identify DoS attacks
Tcpdump is also an excellent tool to help diagnose denial of service (DoS) attacks. These attacks can be somewhat hard to identify, since they normally consist of allowable traffic, just in a large quantity. In addition, bandwidth does not always reach an alarming level in a DoS attack, making it even harder to see what's going on. If you are lucky enough to be passing packets through a Linux box, you can quickly diagnose the problem and seek a resolution. DoS attacks can target just about any service simply by sending enough requests to bog down a server. Those using distributed denial of service (DDoS) attacks can bring even the largest server connected to the largest pipe to its knees.

The first indication of a problem is usually slowness as the server battles to respond to all incoming requests. The particular service targeted will begin to respond slowly and eventually will not respond at all. I have personally seen this attack kill enterprise-level DNS, mail, and Web servers. Step one is to sniff packets associated with the service in question. Running the following command, for instance. will watch all incoming HTTP requests on interface eth0:
$ tcpdump -i eth0 dst port 80

Once you're looking for it, a DoS attack should be fairly easy to diagnose and correct. There will more than likely be an extremely large increase in the number of requests from the same IP address. You can then block that IP address with an access list at the server or router level. A DDoS attack will be a little harder to stop because multiple-source IP addresses will be used, but it is still a fairly simple matter to block the offending addresses.

Once the incoming requests have been blocked, the server should quickly begin to respond as normal. Restarting the process may also be necessary. Then, you have to track down the culprit and make sure that your servers are safe from future attempts, which is usually the hardest part of all.

Tcpdump switches
Let’s take a look at some of the more common switches you may need to use with tcpdump. First is the level of verbosity—how much detail you want to see. Tcpdump provides three levels of verbosity:
  • ·        –v
  • ·        –vv
  • ·        –vvv

The –v switch shows the lowest level of detail, while the –vvv switch shows the most. To specify the verbosity, simply add the switch after the tcpdump command. For the opposite of the –v options, try –q. This will provide quieter output without much detail.

The –w <filename> switch outputs captured information to the specified file. Note that this file will be in binary form. We’ll look at the best way of handling this information in the next section. Another important switch is –i <interface>. This allows you to specify the interface you want to sniff packets on. Examples of acceptable interfaces would be eth0, eth1, and ppp0. The Linux build of tcpdump also provides support for the any switch, which will snoop on all available interfaces, although not in promiscuous mode.

By default, tcpdump looks at only 68 bytes of each packet. This is adequate for most cases, but you can use the –s <bytes> option if you need more. This option allows you to specify the length in bytes to look at in each packet. The tcpdump man page (which is particularly well done) lists NFS and DNS requests as candidates for this switch.

Tcpdump will normally continue to listen until it receives a break. You can manually bypass this requirement by passing it –c <count>. Tcpdump will exit after receiving <count> number of packets that you specify. Another switch, –e, prints link-level headers. On Ethernet, this means the source and destination addresses, protocol, and packet length are printed.

Processing the information
Once you have this wealth of information about your local network activity, what do you do with it? As mentioned above, the –w <filename> switch outputs data in binary form. Tcpdump allows you to process this information with the –r <filename> switch. It will read the file and let you specify switches and arguments as usual. This allows you to capture a snapshot of your network as it is at a certain point, which is great if you need to grab what you can during an intermittent problem. You can always process it later.

Listing A  shows a sample output using the command tcpdump -e -i eth1 dst port 23. In this example, I did not include every line, since there were 43 involved with merely opening a Telnet session from winbox to linuxbox. However, let’s go ahead and break down the first line from the example output.

First, we have the timestamp, 21:16:36.562936, then the source host and port. In this case, winbox is the source host, and the source port is 2168. The destination host and port is linuxbox and telnet. Since 23 is a well-known port, the verbal equivalent (telnet) is provided. The S on the first line indicates that the SYN flag was set. The packet sequence number was 4054180428, and it held no data, which is expressed by (0). There was no piggybacked ack, because this is the first packet, so it had nothing to acknowledge. (However, you can see the ack flags on all the rest of the packets.)

The available receive window is 16384 bytes, and there is a max-segment-size option that requests an MSS of 1460 bytes. The next line is the acknowledgement of the Telnet session, and all the rest are merely the keystrokes as the username and password are entered. The (DF) means that the "do not fragment" bit was set on all the packets.

A number of scripts and programs are available to assist in the interpretation of tcpdump’s output. If you plan to use this program, it is a good idea to also download one of those utilities, such as Sniff. They can aid in diagnosing the output, which, as you can see, can be a somewhat cumbersome task.

Summary
Using tcpdump to inspect packets can be a powerful technique for any network administrator. This program allows you to get detailed information on exactly what is happening at the packet level of your network. Tcpdump can also aid in monitoring and troubleshooting network services on Linux servers. This is important not only for everyday issues but also for serious ones like DoS and DDoS attacks.
0 comments