UNIX/Linux rootkits 101

The term rootkit originated with a reference to the root user account on UNIX systems. Rootkits are not limited to UNIX, however, or even to administrative user accounts such as the UNIX root account. No matter what operating system you use, you should be familiar with good practices for detecting and dealing with the threat of rootkits.

What is a rootkit?

As Mike Mullins explained in his Windows rootkits 101 blog post, rootkits are not exploits. They are not the means of cracking security and accessing your system in the first place. Instead, rootkits are inserted into your system after it has already been compromised for the first time.

Rootkits then cover the malicious security cracker's tracks when he or she revisits the system later or cover the tracks of other malicious software left behind. A rootkit may also include a "back door," allowing the security cracker to gain access at any time in the future.

On UNIX systems such as Solaris or FreeBSD and on UNIX-like systems such as Linux, a number of different means may be employed to cover the security cracker's tracks. Common tactics include replacing system utility binaries such as ls and diff so that when they're used they'll hide changes to the system and files on it from the user.

The key point to keep in mind when dealing with the threat of rootkits is that once a rootkit has been installed on your system, you're no longer able to trust any of the tools installed on that system to give you accurate information. This can make accurate detection of rootkits and other changes to a system by malicious security crackers a challenge.

Rootkit detection

Detecting rootkits on UNIX/Linux systems can involve any of several methodologies:

  • Signature-based detection: Much like antivirus and spyware detection on Windows systems, signature-based rootkit detection uses a scanning tool that checks the system for signs of specific, known rootkits. Signature-based detection is limited because it can't detect new rootkits for which there aren't any available signatures, but it's very easy to implement and use (in contrast with other forms of detection).
  • Heuristic/behavior-based detection: Anomalous behavior of a system can indicate the presence of a rootkit. For instance, unexpected values for reported free space on a hard drive might indicate that the drive contains data that isn't being reported -- data that's hidden by the rootkit. Behavior-based detection like this is possible when a rootkit does an incomplete job of hiding the activities of a malicious security cracker, such as replacing the ls utility to hide specific files but failing to replace the du utility to prevent it from giving an accurate report of files and file sizes in a given directory. The term heuristic is sometimes defined as "prone to error" in a computer science context, however. Rather than using hard-and-fast rules, heuristic and behavior-based rootkit detection involves what amounts to educated guesses, which means that these aren't infallible methods of detecting rootkits.
  • Snapshot/hash-based detection: By creating a snapshot of a filesystem or a part of a filesystem, or by creating an encryption hash of the contents of part of a filesystem, you can later compare that with the current state of the filesystem to determine whether unauthorized changes have been made. This is the simplest of the three concepts of rootkit detection, but it can be the most difficult to actually set up and use because of the sheer number of edge-case conditions that can be expected, including something as basic as accounting for changes in system logs. On the other hand, it's also the methodology that's least prone to error.

The whole point of a rootkit is to hide software that shouldn't be on your system, which obviously includes the rootkit itself. This makes rootkits a uniquely difficult type of malware to detect. A number of tools exist to make this easier for UNIX and Linux users, however, and some of them are available on Windows as well. For instance:

  • The two best-known open source signature-based rootkit detection tools are chkrootkit and Rootkit Hunter. Both of these tools are available in the software management systems of many Linux distributions and free UNIX systems such as FreeBSD.
  • Both chkrootkit and Rootkit Hunter also provide some heuristic rootkit detection capability as well. In addition, there are some commercial heuristic rootkit detection tools for UNIX, just as there are for Windows. Finally, there are a number of different common approaches to admin-defined behavior-based rootkit detection using common system utilities, logfile analysis, and other basic tools of a standard UNIX system.
  • Rootkit detection on UNIX systems via filesystem snapshots and hashes is accomplished by a number of means but more commonly with the Tripwire tool than any other. For complete and secure integrity auditing with Tripwire, as with any other tool, it's best to run your checks from a medium that cannot be accessed and changed from the system being checked to ensure that the integrity auditing tool itself hasn't been compromised. Such tools can also be used for Windows and other operating systems, and there are alternatives to Tripwire such as putting together your own filesystem integrity auditing system with rsync or other backup tools. Even something as simple as a combination of md5 and scp on a separate server can be used to audit filesystem integrity effectively.

Final thoughts

Remember that no system exists in a vacuum. One compromised system may be used to attack other systems that are otherwise less susceptible; learning from the experiences of dealing with rootkits on Windows systems can teach you something about rootkits on UNIX systems and vice versa; any rootkit detection and protection that can be accessed from a compromised system can be compromised by it as well.

Perimeter defense is important, but it's never infallible, and it's not the end of the story. Regular verification that your perimeter defense is working, and discovery of cases where it has not worked, is key to ensuring you minimize any damage that may occur. Detecting rootkits is central to this sort of damage control.

Last but not least, you must have plans in place for what to do if and when your systems are compromised. Protect yourself, recognize when your protection fails, and be ready to deal with the consequences of that failure. Anything less is just asking for trouble. Keep an eye out for a Rootkits 201 post here at TechRepublic's IT Security blog later this week.