For a heavy-duty investigative or forensics tool, look no further than lsof. It comes bundled with pretty much every Linux distribution, and it's an indispensable program. The name of the tool stands for "list open files," and that's exactly what it does, with an amazing amount of detail.
For instance, on a Web server that sends e-mail, has a POP3 and IMAP server, and has MySQL, mailman, etc., executing lsof provides more than 6,200 lines of data. Now, "open files" is a bit of a misnomer as lsof will also display open TCP ports, open UDP ports, open sockets, and so forth. For instance, you could display listening TCP connections and their programs by using the netstat command. The output in Listing A displays open TCP connections.
Now, use the lsof command:
# lsof -i|grep TCP
This command accomplishes the same thing, but you will see much more data output. For instance, with netstat, you see one listening httpd process, but with lsof, you'll see every listening httpd process. To demonstrate more fully, compare these commands:
# netstat -l --tcp -p|grep http
# lsof -i|grep http
With netstat, you'll see two lines of text—one for httpd listening to port 80 and the other for it listening to port 443. With lsof, you'll see every httpd thread listening, which in my case, was 40 processes.
Of course, lsof is useful for more than just tracking which processes are listening to which ports. With it, you can see the programs that are using certain files or filesystems, such as:
# lsof /home
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 14650 joe cwd DIR 9,4 4096 50331821 /home/joe
screen 19868 joe cwd DIR 9,4 4096 50331821 /home/joe
bash 19869 joe cwd DIR 9,4 4096 50331821 /home/joe
ssh 19888 joe cwd DIR 9,4 4096 50331821 /home/joe
Compare this output to fuser:
# fuser -v /home
USER PID ACCESS COMMAND
/home root kernel mount /home
As you can see, lsof can provide a lot more detail than other tools such as fuser or netstat. When investigating problems, be sure to have lsof handy, as it can tell you things that no other program will.
Delivered each Tuesday, TechRepublic's free Linux NetNote provides tips, articles, and other resources to help you hone your Linux skills. Automatically sign up today!
Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.