One of the most frequently used tools for monitoring network connections on a Linux server is Netstat. Netstat returns a variety of information on active connections such as their current status, what hosts are involved, and which programs are involved. You can also see information about the routing table and even get statistics on your network interfaces. Netstat is a good all-around utility and it is an essential tool for the Linux administrator to master.
Netstat is a widely used program and is installed on almost every Linux distribution by default. The program itself can usually be run by any user, so simply typing netstat at a prompt should tell if you if it’s installed. Netstat is part of the Net-tools package, along with such basic programs as ifconfig, arp, hostname, and route. If your system is missing these utilities, you should install them from your Linux distribution CD or download and install them manually.
Netstat offers a variety of options to help you see what network connections are active on your server. Although any user should be able to run the command, only root will see all the possible information.
The output from Netstat is useful in a number of situations. Perhaps you are trying to track down what program is listening on a certain port. Maybe you’ve been playing with inetd.conf and want to see exactly what ports are still open. You can even troubleshoot possible attacks while they are in progress. Listing A shows the output of Netstat when run without any options.
In this example, we see our active network connections. These are made up of ports and the services listening on them as well the protocol in use, the Recv-Q, the Send-Q, the local address, the remote address, and the state. The Recv-Q is the number of bytes not copied by the program connected to the socket and Send-Q is the number of bytes for which no acknowledgment has yet been received.
By default, Netstat will attempt to resolve IP addresses into hostnames. You can shut this off with the –n switch. In the above example, we can see that some connections are being made to the local server. You can quickly tell that we have some SSH and HTTP connections. You can also tell what their current status is by looking at their state. Some of the common states are:
- · LISTEN—The socket is listening for incoming connections. Those sockets are only displayed if the –a or –l switch is set.
- · ESTABLISHED—The socket has an established connection.
- · SYN_SENT—The socket is actively attempting to establish a connection.
- · SYN_RECV—A connection request has been received from the network.
- · TIME_WAIT—The socket is waiting after close to handle packets still in the network.
- · FIN_WAIT1—The socket is closed, and the connection is shutting down.
- · FIN_WAIT2—The connection is closed and the socket is waiting for a shutdown from the remote end.
- · CLOSE_WAIT—The remote end has shut down, and it is waiting for the socket to close.
- · CLOSED—The socket is not being used.
This information is useful in understanding what is happening on a network connection, from seeing which ports are actively listening to viewing incoming connections to examining the exact state of a TCP session. Remember that UDP is not connection oriented, so it will not list states.
All of this can assist in determining what possible problems may exist. A lot of CLOSE_WAIT connections may indicate a port has become unresponsive. A simple kill - HUP command may be enough to revive the process in that case.
You can use a number of optional switches with Netstat. Table A lists some of the most common and useful switches.
|–a||Prints information on all connections, including those in a listening state.|
|–p||Displays the process name and PID of each connection.|
|–n||Shows only numerical addresses; does not resolve IPs to hostnames.|
|–r||Prints the kernel’s routing table.|
|–I||Displays a table of networking interfaces.|
|–s||Shows protocol statistics.|
|–l||Shows just the listening ports.|
|–e||Prints extra information, such as UID and inode.|
Many of these options can be combined with others, depending on what you’re looking for. In some cases, you may want to see the owner of each program that currently is maintaining a network connection; other times, that information is superfluous.
You could run netstat –anp, which would generate a lot of information and would be speeded up by not resolving IP addresses (–n). Showing numerical IP addresses is faster, especially on active servers, and can be especially useful when polling at regular intervals.
It also helps to be familiar with your network topology. For example, say you run Netstat on an internal firewall but see routable IP addresses connected to the host. This could be occurring for a number of reasons, and it would definitely be worth investigating. Knowing what the expected behavior for connections should be can often make the difference between a few minutes of work and a few hours. Spend some time baselining your systems. Then, when something out of the ordinary happens, you’ll be able to quickly identify it and work on pinpointing the root cause.
Listing B shows the results of the netstat –anp command mentioned above.
As you can see, the output now lists sockets that are in a LISTEN state, as well as their PID and program name. We can see an SSH session coming from 10.0.2.142, an HTTPS connection (port 443) waiting to die, and all programs that are listening for connects. And, as expected, the PID and program name are printed at the end of each line.
Netstat is a great starting point when troubleshooting network connection problems and security issues on a Linux server. You can quickly get a list of active interfaces (including errors and other statistics), the kernel routing table, and verbose data on all active network connections. This information can help you find open ports, spot problem daemons, and even identify the source of an attack. My recommendation is to become familiar with the regular output of Netstat on your own Linux servers and then look for irregularities in the Netstat output when problems arise.