Detecting rootkits

You suspect you've been hacked, but how can you be sure? We'll explain how to use cryptographic checksums to validate your bianaries and to snif exactly which ports your systems have been listening to.

By Chris Prosise and Saumil Udayan Shah

In our last column, we examined one of the more common and powerful tools that black-hat hackers use to maintain control over compromised systems: the rootkit. We discussed the key components of the rootkit and showed how it has been used in actual attacks.

This week we revert to the perspective of the system administrator and show how you can detect a rootkit on your system. Rootkit detection is vital and can be one of the more daunting tasks a system administrator faces. In addition to discussing detection, we'll provide preventive steps that you can take immediately, before a rootkit is ever installed.

A note on prevention
A rootkit is not a means of attack but rather a collection of utilities installed and employed by black-hat hackers after they compromise a system is by taking advantage of an existing vulnerability. If you prevent your system from being compromised, you prevent rootkits, so consider the protective measures below. We will examine these measures later in the article.
  • Create MD5 checksums of critical system utilities
  • Record all open ports
  • Save copies of system utilities to floppy or CD-ROM
  • Configure remote syslog hosts
  • Introduce network logging
  • Secure your systems (patch, administer, harden)

Rootkits typically comprise various utilities. In order to detect rootkits, we need to understand their components, as listed here:
  • Trojan horse system utilities that prevent the system administrator from detecting attacker activity
  • Back doors that allow the hacker to enter the system at will
  • Log-wiping utilities that erase the attacker's access record from system log files
  • Packet sniffers that capture network traffic for the attacker
  • Other utilities that can be used for communication or further attacks

Trojan horse system utilities
Trojan horse system utilities are usually diagnostic tools such as netstat, ps, df, find, and ls. Attackers modify these utilities so that system administrators can't identify the attackers' files, processes, or activities. Attackers are very crafty with their Trojan horse files—the size and time/date stamp of the file are often identical to those of the original, which makes detection difficult. The easiest way to detect if these binaries have been sabotaged is through the use of cryptographic checksums. A cryptographic checksum is a digital snapshot of a file. No two files should have the same snapshot, or checksum. Use the MD5 algorithm to compare the checksums of your binaries to the checksums of known good binaries.

Your library of safe MD5 checksums should be calculated from sources that the attacker cannot also modify, either original CD-ROMs or checksums direct from vendors via the Web. Also—and this is crucial—when calculating the checksum of suspected Trojan horse binaries, do not use a copy of the MD5 program that is on the victim system. What if the crafty attacker also sabotaged the MD5 program to display incorrect checksums for given binaries? You should use a known good copy of MD5 from media such as CD-ROM or floppy diskette instead. In fact, as part of normal system administration duties, it is a great idea to save a known good copy of all important system utilities to a CD-ROM or floppy. This way, in the event of an incident, you can use the good saved copies of the binaries to investigate. You can also automate the process of creating and comparing checksums with Tripwire.

Back doors
Back doors are usually programs that listen on a TCP or UDP port and allow an attacker remote access to the system. In the case of the rootkit we examined in the last column, the back door program, bd2, was a remote procedure call service. To determine if your system has any listening UDP or TCP ports, use a portscanner such as Nmap or FScan. Here's an example of the results from running FScan:
C:>fscan -p 1-65535 -u 1-65535
FScan v1.12 - Command line port scanner.
Copyright 2000 (c) by Foundstone, Inc.

   Scan started at Tue Jan 23 05:16:44 2001 25/tcp 110/tcp 8001/tcp 161/udp

   Scan finished at Tue Jan 23 05:27:01 2001
   Time taken: 131070 ports in 617.017 secs (106.22 ports/sec)

Once again, remember to use a known good copy of scanning software if you scan the suspected victim system locally, because an attacker could have modified scanning software on the victim system.

Any open port is a potential back door and should be investigated. To identify the binary associated with each open port, use a known good copy of the netstat command with syntax netstat -anp. This will display all connections and listening ports with addresses in numerical form. The p option will also display the program associated with each listening port. Investigate each program to ensure legitimacy. If something like bd2 appears, your job was easy. However, attackers may obfuscate the names of programs, or even modify existing programs such as ftpd. In this case, investigate the program using tools such as MD5 and strings.

Log-wiping utilities
In our last column, we saw that the black-hat hackers erased their tracks from logfiles such as utmp, wtmp, messages, and syslog. For that reason, even if you reviewed the host logs regularly, you would not notice the compromise. However, with a little preparation, you can create copies of logfiles that cannot be modified by attackers. Syslog messages can be sent to any logfile of your choice and, more importantly, any host of your choice. Consider what happens when a hacker compromises a host that logs messages to a remote server, then uses a rootkit utility. Because syslog messages are created and sent in real time, the rootkit utility deletes only the log entries on the local host.

Virtually any host can be configured to receive syslog messages. We recommend a secure, central syslog server that collects logs from all critical hosts. Unix systems are easily configured to log messages to remote hosts using the syslog configuration file, syslog.conf. The following entry logs all auth messages (generally security related) to the host
auth.*         @

Packet sniffers
Virtually every rootkit includes utilities for sniffing network traffic. Detecting network sniffers is difficult but not impossible. On several Unix flavors, the ifconfig command allows the privileged administrator to determine whether any interfaces are in promiscuous mode. An interface in promiscuous mode is listening to all network traffic, a key indicator that a network sniffer is being used. To check your interfaces using ifconfig, just type ifconfig -a and look for the string PROMISC. If this string exists, your interface is in promiscuous mode and you will need to investigate further, using tools such as ps to identify the appropriate process.

Alternatively, you can use AntiSniff to remotely detect network cards in promiscuous mode. This tool uses several techniques, some of which are more reliable than others. Although the tool is useful, it is not reliable in every case and should not be considered authoritative.

Other rootkit utilities
Rootkits may contain tools that serve a variety of purposes, such as communication channels or distributed denial of service agents. The example in our last column had both of these programs, which could be detected through network monitoring. Although network intrusion detection systems are an excellent resource, it is also important to have the ability to capture all traffic. If a rootkit is suspected, one of your first steps should be to monitor all network traffic to and from the suspected victim system. Using a computer on the same network segment as the victim system, employ a tool such as tcpdump to log all traffic; just the TCP and IP header information is sufficient. Examine the traffic to determine if unusual traffic is being sent. For example, thousands of outgoing UDP and ICMP packets to unauthorized hosts would probably signify a distributed denial of service attack.

We hope we've provided some insight into rootkits and given you useful information for protecting your networks. As with many complex problems, rootkits are more manageable when the components are analyzed separately. Steps taken now, such as remote system logging and saving known good copies of system utilities, will be useful in the event that you suspect a rootkit has been installed on your server. Investigative steps such as network monitoring and cryptographic checksums will provide the backbone of actual detection. As with most security issues, an ounce of prevention is worth a pound of cure. Do you have other useful techniques for detecting rootkits? We welcome all strategies at


Chris Prosise is the vice president of professional services at Foundstone, a network security firm specializing in consulting and training. Formerly a U.S. Air Force officer and a Big 5 consultant, Chris is the coauthor of Incident Response: Investigating Computer Crime and is an adjunct professor at Carnegie Mellon University. Chris holds a B.S. in electrical engineering from Duke University and is a Certified Information Systems Security Professional (CISSP).


Saumil Udayan Shah, principal consultant for Foundstone, provides information security consulting services to Foundstone clients. Shah specializes in ethical hacking and security architecture. He holds an M.S. in computer science from Purdue University and is a Certified Information Systems Security Professional (CISSP).

Editor's Picks

Free Newsletters, In your Inbox