If the thought of scanning through log files for a few hours each day is not at all appealing to you, consider two very useful tools. The first is Logcheck, a program that scans your log files and sends a report to the administrator based on specific guidelines. Logcheck can be configured to send you a message each day with the relevant entries of any suspicious information recorded to your log files. The other program is swatch, and it's a Perl program that also checks your log files but offers a little more flexibility.
In this Daily Drill Down, we'll take a look at both programs so you can decide which is better for you. Both are excellent programs with a lot of automated help to offer.
Why use log files?
When it comes to securing your system, there are a few steps that need to be taken and a few layers of security that can be implemented. The level of detail that you use will determine how secure it is and how quickly you can correct any potential problems.
Using a firewall is a good method to protect your computer or network. Tools such as nmap and Nessus are excellent for tracking down potential holes in your system and giving you the information you need to seal them, as well as making you aware of the status of your system. These are excellent starting points, but is the job done once you have used and configured them? Not even close. Once you have configured your first line of defense and analyzed it, you have more work ahead of you than you did before.
Your system, whether it is a firewall or a computer behind one, generates an insane amount of logged information. This information contains pretty much everything that is happening on your system. It can be used to diagnose a variety of problems on your system, including potential and successful hack attempts.
For this reason, it is recommended that your log files be scrutinized daily, if not on a constant basis. But with the amount of logging any Linux system produces, you could spend most of your day looking at log files. Not only is this unproductive (in terms of your other work), but it's extremely tedious and, quite frankly, very boring.
It is important that the job get done, however. Log files exist for analysis, and if they aren't being looked at, an essential resource is being wasted. Also, you are seriously reducing your ability to counter any potential attacks.
Logcheck by Psionic Software is a log-file-auditing program released under the GPL. It is available at the Psionic Web site for free download. It was originally meant as a means of processing the log files for the Abacus Project tools and a few other security tools, like TCP Wrappers and the Firewall Toolkit. However, Logcheck is configurable enough to be used as an auditing tool for general log files under various Linux, *NIX, and BSD operating systems.
Logcheck uses a single program called logtail to scan the log files. A wrapper script called logcheck.sh is used to call logtail against the log files with criteria defined in special configuration files. The logcheck.sh program is invoked daily, usually by cron, to scan your log files and report any suspicious activity to a defined user.
To install Logcheck, download the latest version, which is 1.1.1 as of this writing. The file to download is called logcheck-1.1.1.tar.gz and should be saved in your /usr/local/src directory.
Change to that directory and unarchive the file using:
tar xvzf logcheck-1.1.1.tar.gz
This creates a subdirectory called logcheck-1.1.1, so change into it. You will need to run the following command to compile logtail and install the programs and configuration files:
When this is done, the logtail binary will be installed into /usr/local/bin, while logtail.sh and the Logcheck configuration files will be installed in the /usr/local/etc directory. If you wish to change the location of the installed files, you must edit the Makefile contained in the logcheck-1.1.1 directory and change the value of INSTALLDIR to where you want the configuration files to go, the value of INSTALLDIR_BIN to where you want logtail to be installed, and INSTALLDIR_SH to where you want logcheck.sh to be installed. For instance, if you wanted the two programs in /usr/bin and the configuration files in /etc/logcheck, you would assign the following values in the Makefile:
INSTALLDIR = /etc/logcheck
INSTALLDIR_BIN = /usr/bin
INSTALLDIR_SH = /usr/bin
You can also change the temporary directory that Logcheck uses from the default /usr/local/etc/tmp to something like /var/logcheck, using
TMPDIR = /var/logcheck
in the Makefile as well.
Once this is complete and you have the files where you want them, you should edit your logcheck.sh script and change a few values. The primary field to edit is the value of SYSADMIN in the script, which is the user to whom the Logcheck reports are sent. By default this is root, so you might want to change this to your username. For instance, to send the e-mail reports to the user joe, you would use
If you change the default locations of the temporary directory and the logtail program, you should also change the LOGTAIL and TMPDIR values to reflect those changes.
Logcheck uses configuration files to search for strings in the log file. These configuration files are logcheck.hacking, logcheck.violations, logcheck.violations.ignore, and logcheck.ignore. You should take a look at these files as well to fine-tune the strings that Logcheck matches against, as by default it is pretty paranoid. By default, these files are installed in /usr/local/etc. Each file contains different strings and is categorized in a specific manner. The logcheck.hacking file contains active hacking messages to look for. It can sometimes cause false alarms because it is a rather generic way of looking at the log files, but it also reduces the amount of logging text and time spent looking at it, so a few false alarms are better than nothing. This file can only be accurate if you have previous logging messages to compare it to. By default, it looks for generic ISS probes and obvious sendmail attacks and probes.
The logcheck.violations file contains patterns to specifically look for that would indicate a security violation. This file contains keywords that are typically negative (such as "refused" or "denied") and which may indicate some sort of attack, failed or otherwise.
The logcheck.violations.ignore file contains more complete sentences that have keywords from the violations file. The default installed file is usually good enough, but you can fine-tune it if you need to. Keep in mind that you must not leave this file empty.
Finally, the logcheck.ignore file contains patterns that should be ignored if found in a log file. If you have repeated false alarms in your e-mail messages, you can put a matching pattern here to prevent the false alarms from repeating.
The defaults for these four files should be sufficient for most situations, but may require fine-tuning for systems that do not use wu-ftpd or sendmail. Note also that you can use standard wildcards in your pattern matches, as these are simply regular expressions to match in the log files.
By default, Logcheck will scan three files under Linux: /var/log/messages, /var/log/secure, and /var/log/maillog. If you want to add more logs for Logcheck to scan, you must add them in the logcheck.sh script where these commands are called. In an unmodified logcheck.sh, under Linux, these start on line 156.
Finally, you can add the following entry to your root user's crontab to execute Logcheck hourly:
01 * * * * /usr/local/etc/logcheck.sh
or to run it daily at 4:02 A.M.:
02 4 * * * /usr/local/etc/logcheck.sh
You may choose to run Logcheck once a day so you can have an overview of what happened the previous day. This does not alert you to active attacks at the time they occur, but it does let you know the status of your system. If you run Logcheck hourly, you should expect an e-mail message every hour.
Where Logcheck simply reports what it finds in log files by sending an e-mail, the swatch program (also known as the Simple WATCHdog) is a little more flexible and intuitive. It can be used to send e-mail messages, execute a specified script, and so forth if it comes across a string in a log file that it is configured to be monitoring. Swatch is a Perl program written by Todd Atkins and is available from his Stanford Web site. The file to download is swatch.tar or latest.tar. Download either of these files to your /usr/local/src directory and unarchive it using:
tar xvf swatch.tar
The current version of swatch is 3.0.1, and the subdirectory that contains the program also has the version number in it, so you would change to the swatch-3.0.1 subdirectory. Because swatch is a Perl program, you will need to compile it a little differently by executing the following commands:
This will install two files in /usr/bin, as well as the swatch and swatch_oldrc2newrc programs. It will also install two manfiles, one for each program.
In the examples/ subdirectory you will find some example rc files, or configuration files, for swatch. The configuration file is very simple and straightforward. You can define a number of "watchfor" keywords, which indicate strings or regular expressions to check for in your log files. After each "watchfor" keyword and subsequent string, you can define the actions that swatch is to execute if that string is matched. Let's take a look at a few examples:
# Bad login attempts
exec "/usr/local/sbin/badloginfinger $0"
This tells swatch to look for the strings "INVALID,""REPEATED," or "INCOMPLETE." If any of these are matched, swatch will take the actions specified. In this case, swatch will echo the matched line to the screen, which is useful if you are running swatch in its own xterm. Next, it will ring a bell through your PC speaker three times and, finally, it will execute the command /usr/local/sbin/badloginfinger $0 which executes the badloginfinger program with the entire matched log line as input.
Swatch knows there are no more actions to run for this matched string once it encounters another "watchfor" keyword or the end of the file. There are a number of other useful commands that can be used, like the mail command, which follows the syntax
If you wanted to send an e-mail to email@example.com and firstname.lastname@example.org with the subject "Hack attempt!" you would use:
The reason for writing the e-mail addresses using \@ instead of just @ is because swatch is a Perl program and Perl requires that all @ characters that are used literally be escaped with the backslash character.
The throttle command is used to limit the number of times that the matched pattern has actions performed on it. So instead of receiving 20 messages within 20 seconds because of a repeated probe (or any other repeated message), you could use the throttle command like this:
This will prevent multiple instances of the matched string from being acted upon if they are received within one minute of the first matched string.
The "watchfor" regular expression must be enclosed with forward slashes like this:
If the regular expression is not written between two forward slashes, swatch will not operate properly and you will not get the results you expect.
Running swatch is quite simple as well. Swatch takes a number of command-line arguments with a variety of functions. Typically, you will call swatch in this manner:
swatch --config-file=/etc/swatch/swatchrc --tail-file=/var/log/messages
This tells swatch to use the configuration file /etc/swatch/swatchrc and to monitor the /var/log/messages log file. You can also have swatch watch named pipes or examine a single log file with a single pass by using the --read-pipe and --examine command-line options. If you do not tell swatch what configuration file to use, it will use ~/.swatchrc by default.
Contrary to Logcheck's method of being executed once an hour or day by cron, the ideal way to run swatch is to start it as a background process once you boot your computer. This allows swatch to monitor a log file all the time and alert you in real time as it matches your defined regular expressions. You could run swatch once a day using the --examine command-line option, but you run the risk of duplicate messages being reported if you do not rotate your logs daily.
Ideally, I suggest adding the following to your /etc/rc.d/rc.local file:
The above command is actually one line, which has been broken in two for readability.
This will start swatch each time your system boots up. This also allows you to receive your alerts in a more real-time fashion than with any other log-auditing package.
You must also have syslog running and optimized for use with something like swatch. The benefit of Logcheck is that it can easily check multiple log files. With swatch, however, if you want it to monitor more than one log file, you will need to run multiple instances of swatch, which may not be very efficient. The alternative is to tell syslog to log everything to one single file rather than the standard log files and tell swatch to monitor this file instead.
On most Linux distributions, the /var/log/messages file is the most comprehensive log file and might be all that you need to audit. The only things that most distributions do not log to this file are private authentication messages and mail or news messages. Since we may want these messages, we would add a new entry to our /etc/syslog.conf file, like this:
This will log everything of facility info or higher to the file /var/log/swatch. A word of warning: This file will get very big very fast. Take a look at the size of your /var/log/messages file and realize that this file will get even bigger than that. So now, instead of telling swatch to monitor /var/log/messages, you would tell it to monitor /var/log/swatch.
Additionally, it might be ideal in a network situation to create one system as a master logger for your network. By using syslog's ability to write log messages both locally and remotely, you can use Logcheck or swatch on the "logmaster" system to alert you of problems across your entire network without the need for setting up the same system on each machine. This setup has a number of advantages. The first is security for your log files. If someone manages to compromise one of your computers, they will most likely attempt to scrub the log files after they are finished to remove any traces that they were there. By using an active log monitoring system like swatch, you can become aware of the problem before they have a chance to alter your log files. And by writing all of your logs to a single "logmaster" system, you can retain copies of the log messages even if the intruder does manage to scrub the logs on the compromised host.
By using swatch on the "logmaster" system, you can also have it running in an xterm on your desktop monitoring a master log file so you can see exactly what is happening across your network in real time. This will alert you to possible crack attempts and other situations that may arise on the network, like system reboots.
In the end, the best offense is a good defense. By using a program like swatch or Logcheck, you are strengthening your defense and making it more difficult for crackers to gain unauthorized access to your system. With the versatility of these programs, you can seriously reduce successful attempts to compromise your systems by stopping an attacker before they can complete the task.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.
Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.