Utilities help you detect and clean rootkits from a Linux box

Rootkits have been the "thing under the bed" for Linux administrators for a number of years. Now there are tools to aid in the detection of these monsters. Scott Lowe shows you how to use these utilities and how best to clean up after the fact.

Rootkits are particularly dangerous utilities installed by an attacker to thwart unknowing systems administrators and take control of system resources. To make matters worse, rootkits have a lot of built-in tricks that make them hard to detect once they’ve been installed. From clearing log files to exactly matching file sizes and checksums, rootkits are deft at masking their signature.

In this Daily Drill Down, I’ll discuss two tools that help determine whether or not a rootkit has been installed onto a system. I’ll also show you how to rid the system of the havoc rootkits can leave behind.

More information
For more information on rootkits, take a look at my article “Rootkit threats move beyond Linux to Windows systems.”

It’s the little differences
Even though rootkits are difficult to detect, doing so is not impossible. In fact, some rootkits unwittingly leave subtle hints that there’s something amiss. For example, with certain versions of Linux rootkits, the hacked login utility does not act in exactly the same way as the clean version. One subtle difference is that the login process on a compromised system is somewhat slower than normal.

Even though this may seem like a minor detail, it illustrates the importance of watching subtle changes that might occur on a Linux server and responding to anything that seems unusual. It also highlights the need for effective tools to hunt down and weed out rootkits, and the open-source community has responded with tools expressly for that purpose.

One of the most popular rootkit detection programs is called chkrootkit and is currently at version 0.35, which was released on Jan. 17, 2002. The chkrootkit install includes a number of C programs that check for various breadcrumbs left behind by rootkit installers.

One of these C programs checks to make sure that network interfaces are not in promiscuous mode, which would suggest that a sniffer is running. You might ask why you can’t just check the output from the ifconfig utility to do this. Most rootkits will modify the ifconfig program to prevent this status from being displayed. They’re tricky.

The chkrootkit program checks many commonly modified binary programs for modifications, including sshd, rshd, telnet, and top. It can detect well over 30 rootkits, including t0rn, lrk3, lrk4, lrk5, lrk6, and the Lion worm. Also included in chkrootkit are the chklastlog, chkwtmp, and check_wtmpxutilities, which detect deletions in various log files.

Installing chkrootkit
This invaluable utility is available for Linux 2.0, 2.2, and 2.4, FreeBSD 2.2, 3, and 4, OpenBSD 2.6 – 2.9, and Solaris 2.5.1, 2.6, and 8. Installation is very simple. The downloaded file name is chkrootkit.tar.gz, and I have downloaded it onto a suspect system into my home directory at /home/slowe. To install chkrootkit, I used the following commands:
cd /home/slowe
gunzip –dc chkrootkit.tar.gz | tar xvf –
cd chkrootkit-0.35
make sense

Execution of the chkrootkit command results in the output shown in Listing A, which in this example gave me evidence that my system was not infected. As you can see, chkrootkit is quite extensive. Notice that many utilities are tested and the Ethernet interface is verified to be running normally and not in promiscuous mode.

To prove that chkrootkit actually works, I replaced the netstat utility with a version that I know to be from a rootkit and then re-ran chkrootkit. Listing B shows the portion of the chkrootkit output that indicates there’s a problem.

chkrootkit best practices
To function properly,chkrootkit depends on certain system binaries, including netstat, find, and ps. Since these binaries are generally modified by a rootkit, you should use binaries that you know to be good, such as those burned onto a CD-ROM or from distribution CDs. chkrootkit can use such known good binaries by using the ‘-p’ option on the command line:
./chkconfig –p /path/to/binaries

Kernel Security Therapy Anti Trolls, or KSTAT, is designed to verify the status of the kernel, which helps aid in the detection of a loadable kernel module (LKM) rootkit. LKM rootkits are particularly difficult to detect due to their tight integration with the system and the fact that they intercept system calls from unmodified system binaries. Simply checking the MD5 checksum of a system binary will not detect this type of rootkit. KSTAT can be downloaded as kstat.tgz and installed as follows:
gunzip –dc kstat.tgz | tar xvf –

Issuing a command “./kstat –s” against a system that is infected with an LKM rootkit will yield the results similar to those shown in Listing C.

The warnings indicate that the particular call has been modified and should be treated with caution. Many of these calls are used internally by non-Trojan versions of system utilities, so this information may be all that you need to detect the existence of a rootkit.

The Web site listed for this utility includes several other utilities for FreeBSD and Linux, as well as an in-depth explanation of LKM rootkits and how they work. It’s a very interesting read.

Cleaning the system
So you’ve run both KSTAT and chkrootkit and found that someone has violated your system’s integrity and installed a rootkit. What should you do? First of all, take it very seriously and assume the worst. Your system may have been compromised for quite some time, and the attacker may have installed any number of rootkits and other backdoors.

A common practice, and likely the safest long-term option once a rootkit or other major system compromise has been detected, is to erase the entire system and reload it from installation media and backups. Unfortunately, this is not always a viable option, so you’ll need ways to clean up the mess rather than simply start over.

Using the chkrootkit utility as described above can be a major help in any clean-up effort. If you look back at the utility’s detection output, you’ll notice that chkrootkit identifies suspect system binaries with an “INFECTED” status. Each affected binary will need to be replaced with a known uninfected version of the file. In the case of my afflicted netstat binary, I need to reload it from the Red Hat Linux 7.2 distribution CD and run chkrootkit again just to make sure that it has been replaced properly.

While replacing binaries sounds simple enough, cleaning up after a rootkit can be a major hassle. Any backdoor users or Trojans that were created and installed by the hacker need to be addressed. Replacing the binaries will go a long way toward achieving this goal, but any backdoor logins that were created need to be removed manually.

Finding these logins is not a job for chkrootkit, however. Although a great utility, chkrootkit is limited to finding information that it knows to be corrupt, such as certain signatures in the system binaries that indicate they are Trojans. Finding suspect logins requires some detective work, and you can’t depend on any of the system logs since they were likely compromised during the rootkit installation. The only way you’ll be able to find suspect logins is via a manual inspection of the user database, which you can perform with the userconf tool, Linuxconf, or by manually browsing the directory hierarchy for any suspected new user directories.

If the rootkit is designed to attack the Linux kernel, it’s much more difficult to clean up after. These types of rootkits replace /proc, where most system calls are made. Recovering from this attack means replacing /proc from a valid backup or another source.

How did they do it?
Most importantly, before you completely clean your system, you need to determine exactly how the attacker gained access and then close the door. In the heat of the moment, this is an often overlooked step. If you skip it and clean the system by reverting it to its original form, the same vulnerabilities will be present. The attacker need only to exploit the same one that was used to gain entry in the first place.

There are too many potential exploits to list here, but keeping your system patched and paying strict attention to CERT advisories is one way to begin your investigation of how the attacker gained entry.

If you can’t determine how the attacker gained entry, you might want to lay a trap by actually leaving the rootkit installed and waiting for the attacker to come back. But do this only after you have installed keylogging software, such as Uberkey, and hidden it from detection. Using this method can help you to determine what route the attacker uses to gain entry. For example, if the attacker is able to log in as rewt with a password of satori, it is likely that a variant of lrk (the Linux Root Kit) has been installed and the attacker is exploiting a trojaned /bin/login to log in to the system again.

Common password
The use of the satori password is common across all versions of lrk’s infected binaries.

Never enough help
You can find some excellent resources on the Web when you find that you have been “owned,” as hackers like to call a successful attack. The SANS site offers up-to-date statistics on attacks worldwide, as well as valuable information about specific rootkits. Another great resource is RootPrompt. Again, it includes a great deal of information on specific rootkits.

Due to the sheer number of rootkits and variants, these sites are essential tools when attempting to recover from an attack or when trying to get specific information about a particular rootkit.

Editor's Picks

Free Newsletters, In your Inbox