Developer

Default installation directories

Log files are critical for the Linux system administrator. Unfortunately, many view using log files as a painful and arduous task. With the aid of swatch and logcheck, the monitoring of many critical systems soon becomes painless.


In a previous Daily Drill Down, “Managing Linux log files,” I discussed the logrotate program and showed how it is used to make sure system log files are rotated and backed up. In this Daily Drill Down, I will look at the swatch and logcheck programs and show how they are employed on a Linux system to monitor system log files. The topics we will cover include:
  • Installing and configuring swatch and logcheck.
  • Deciding which patterns and expression swatch and logcheck should search for and which they should ignore.
  • Running swatch and logcheck as cronjobs.

Using swatch
The swatch (the simple watchdog) program is used to monitor system log files and notify the administrator when events occur. It may be run in either of the following modes:
  • Batch mode: Where swatch examines the contents of a log file and exits
  • Monitor mode: Where swatch continuously monitors log files and evaluates each new entry to the log file by using the tail command

While swatch may provide almost real-time information, it is necessary to have a swatch process running on each server or workstation you want to monitor. An additional swatch process is necessary for each log file that requires monitoring.

Obtaining and installing swatch
The swatch program may be obtained from a Purdue University FTP site in the form of the following three files:
  • FILES
  • Swatch.tar.asc
  • Swatch-3.05b.tar.gz

After you’ve downloaded the files, run the following command to verify the integrity of the swatch package:
pgp swatch.tar.gz
The swatch program requires Perl 5.0 and the following Perl modules: Time::HiRes, Date::Calc, and File::Tail.
The swatch installation script will try to install the required Perl modules from the Comprehensive Perl Archive Network. The swatch program is installed by running the command
Perl Makefile.pl

If you receive an error message stating “swatch missing” or “core dump,” change the following line in /usr/bin/swatch:
#1/bin/sh
eval `exec perl $0 $(1+''$@''}'


to
#!/usr/bin/perl
eval `exec perl $0 $(1+''$@''}'

The /bin/sh line will not work on Linux systems.
Configuring and running swatch
The swatch program reads a configuration to determine which events are to be reported and which methods to use to report these events. It may also be run from the command line, using any of several options to pass instructions to swatch. If no options are specified, swatch is run with these options:
Swatch -config-file=~/.swatchrc -tail-file=/var/log/messages

Table 1 provides a complete list of command-line options available to swatch.

Table 1
Option
Use
—config-file=<filename>-c <filename> This specifies the path to the swatch configuration file. The default is ~/.swatchrc.
—restart-time=[+]hh:mm[am | pm] -r [+]hh:mm[am | pm] This sets swatch to restart after hh hours and mm minutes. A 24-hour clock is assumed if am or pm is not specified.
—input-record-separator=regular_expression This sets regular_expression as the record
separator. The default is a carriage return.
—tail-file=<filename>-t <filename> This allows the filename specified to be continuously monitored as new entries are added, but swatch will not exit when this option is used.
Read-pipe=<command>-p <command> The output of the specified command is
run through swatch.
—examine=<filename>-f <filename> The contents of the specified file are run through swatch. After examining the file, swatch will exit.
Command-line options available to swatch

The swatch configuration file
The swatch configuration file contains a list of patterns along with actions to be taken when that pattern is found. The pattern must be a regular expression that can be parsed by Perl. Table 2 lists the keywords swatch uses to establish which patterns to look for and what action(s) to take when these patterns are found.

Table 2
Keyword
Function
watchfor=<regular_expression> This specifies the regular expression to look for.
ignore=<regular_expression> This specifies which regular expression to ignore.
waitfor=<regular_expression> This tells swatch to ignore all regular expression until this one is found.
Echo [=mode] This outputs the line containing the matched regular expression.
Bell = [n] When a regular expression is matched, the terminal bell rings n times, and the line is output.
exec=<command> The specified command is executed when a regular expression is matched.
mail= <address1 : address2> The matched line is mailed to the address specified. If no address is specified, the line is mailed to the user running swatch, usually root.
Pipe= <command> The matched lines are piped to the specified command.
write= [user1:user2] The matched line is echoed to the specified user(s), using the write utility.
throttle=options This is used to limit the actions taken on a matched line. The specified actions are taken the first time a regular expression is matched, and then the options set the amount of time to wait before taking additional actions.
Keywords available to swatch

How to use swatch on your system
The swatch program includes some sample configuration files. Click here to view the code showing a configuration file that may be used to monitor /var/log/messages.
#
# Personal Swatch configuration file
#

# Alert the administrator of bad login attempts and find out who is on that system
watchfor /INVALID|REPEATED|INCOMPLETE/
  echo inverse
exec=''/usr/local/sbin/call_pager.pl 9021212 555''
exec=''/usr/local/sbin/badloginfinger.pl $0''
  bell 3


# Important program errors
watchfor /LOGIN/
  echo inverse
  bell 3
watchfor /passwd/
  echo bold
  bell 3
watchfor /ruserok/
  echo bold
  bell 3

# Ignore this stuff
ignore /sendmail/,/nntp/,/xntp|ntpd/,/faxspooler/

# Report unusual tftp info
ignore /tftpd.*(ncd|kfps|normal exit)/
watchfor /tftpd/
  echo
  bell 3
# Kernel problems
watchfor /file system full/
  echo bold
  bell 3
ignore /vmunix.*(at|on)/
watchfor /vmunix/
  echo
  bell
watchfor /fingerd.*(root|[Tt]ip|guest)/
  echo
  bell 3
watchfor /su:/
  echo bold
exec=''/usr/local/sbin/call_pager.pl 5551212 333''
mail=root
watchfor /.*/
  echo

Methods employed by the administrator in this example include the following:
  • The swatch program reports only what it is configured to report, so it may want to keep the pattern
    watchfor /.*/
    at the end of this file. As you become more familiar with the monitoring process, you can add patterns for swatch to ignore.
  • The line
    exec=''/usr/local/sbin/call_pager.pl 9021212 555''
    is used to provide the administrator with real-time notification of bad login attempts. When a bad login is attempted, the administrator will receive the code 555 on his pager.
  • The line
    Exec=''/usr/local/sbin/badloginfinger.pl
    is used to safely finger the user to make the login attempt.
  • The line
    exec=/usr/local/sbin/call_pager.pl 9021212 333''
    sends the code 333 to the administrator via his/her pager at 902-1212 (this number represents our example above) if an attempt is made to execute the su command.

The exec command is typically used to run a command that will provide the administrator with real-time notification of a problem. The call_pager.pl and badloginfinger.pl are two Perl scripts that provide this functionality to swatch.

Restarting swatch
Log files are often rotated on a daily basis. When these files are rotated, the administrator will want to restart swatch. It may also be necessary to restart swatch if the configuration file(s) are edited while swatch is running.

To restart swatch at 5:00 P.M. every day, after log files are rotated, the administrator may use either of the following commands. The first command is entered at the command prompt and is a direct invocation of the swatch program:
Swatch -t /var/log/messages -c /etc/swatch/swatchrc.messages -r 5:00

The next command involves restarting swatch through an HUP signal, issued through the /etc/logrotate.d/messages file. Click here to view the code.
/var/log/messages {
prerotate
/usr/bin/chattr -a /var/log/messages
endscript
postrotate
/usr/bin/killall -HUP syslogd
/usr/bin/chattr +a /vat/log/messages
/usr/sbin/swatch -t /var/tail/messages -c /usr/local/etc/swatch/swatchrc.messages -r 5:00
endscript
}

The swatch configuration file may be updated while swatch is running. When this happens, use the command
swatch - HUP

to force swatch to restart with the updated configuration file.

To terminate swatch correctly, use a QUIT, TERM, or INT signal. The swatch program is but one tool available to system administrators. It works well, but it should be employed as part of a suite of security tools on a Linux system.

Using logcheck
Logcheck is used to check new log files for unusual entries. The administrator is then mailed a summary of these events. Logcheck works by matching keywords in a configuration file against events taking place on your system. Logcheck will report any event that it has not been explicitly told to ignore.

While swatch is used for real-time notification of system events, logcheck is typically used to report log entries that may not be time-sensitive but are still very important.

Obtaining and installing logcheck
Logcheck may also be downloaded from Purdue University.

To ensure that you have a secure version of logcheck, download the following files:
  • logcheck-1.1.tar.gz
  • logcheck-1.1.tar.gz.asc
  • signature-file.tar.gz

Next, run the following commands to verify the integrity of the downloaded package:
pgp signature-file.tar.gz
pgp logcheck-1.1.tar.gz.asc


Then, run the command
tar -zxvf logcheck-1.1.tar.gz

to decompress and untar the logcheck package in the directory of your choice.

Configuring logcheck
The first step in configuring logcheck is to ensure that logcheck mails its output to the correct person. This is accomplished by going to the logcheck-1.1 directory and editing the file
/systems/linux/logcheck.sh

Look for the CONFIGURATION SECTION in this file, and confirm that the SYSADMIN variable is set to the correct user. The default is root, so unless you want logcheck to mail its output to another user, this section may be left as is.

The next step is to choose the default installation directories. Click here to see the defaults.
LOGTAIL=/usr/local/bin/logtail
TMPDIR=/usr/local/etc/tmp
GREP=egrep
MAIL=mail
HACKING_FILE=/usr/local/etc/logcheck.hacking
VIOLATIONS_FILE=/usr/local/etc/logcheck.violations
VIOLAIONS_IGNORE_FILE=/usr/local/etc/logcheck.violations.ignore
IGNORE_FILE=/usr/local/etc/logcheck.ignore

The defaults are fine, but the administrator may choose any directory in which to place these files.

The next step is to confirm the files you want logcheck to monitor. The defaults are:
$LOGTAIL /var/log/messages > $TMPDIR/check.$$
$LOGTAIL /var/log/secure > > $TMPDIR/check.$$
$LOGTAIL /var/log/maillog > > $TMPDIR/check.$$
$LOGTAIL /var/log/tcplog > > $TMPDIR/check.$$


The administrator may add any files to this list that they would like to have logcheck monitor.

Now, to install logcheck, as root, run the command:
make linux

Once logcheck is installed, set up a cronjob to run logcheck every hour, as shown here:
0 **** /usr/local/etc/logcheck.sh

How logcheck works
Logcheck looks for patterns by reading four files. These four files are read in order and may be edited to add or remove patterns. The four files and their functions are listed in Table 3.

Table 3
File
Function
logcheck.hacking This is the first file read by logcheck. The logcheck.hacking file contains keywords commonly used during system attacks. Any match for these keywords is shown at the top of the log file report. The search for these keywords is not case-sensitive.
logcheck.violations The second configuration file read is logcheck.violations.
A match for a pattern contained in this file may indicate
either a system problem or a security threat. The search
for patterns in this file is not case-sensitive.
logcheck.violations.ignore This is the third file read by logcheck. This file contains keywords that logcheck will ignore. The search for keywords contained in log.violations.ignore is case-sensitive.
logcheck.ignore The logcheck.ignore is the last configuration file read
by logcheck. The keywords contained in this file will be
ignored when they are matched by logcheck. The search
for keywords contained in this file is case-sensitive. Be
very careful when telling logcheck which patterns to ignore.
Specific files (and their functions) used by logcheck

Running logcheck
The first time logcheck is run, the following events occur:
  • Logcheck will read all of the log files it was configured to monitor.
  • Logcheck will then create a logfile.offset file in the same directory as the log file being read.
  • All events in all log files read by logcheck are then parsed (read) and compared to patterns in the logcheck configuration files.
  • All events that are not ignored are mailed to the user specified in the logcheck.sh file.
  • Whenever logcheck is run after this point, the logfile.offset file is read and any part of the log file that has already been processed will be ignored.

Troubleshooting logcheck
If logcheck reports are not being received, take the following steps:
  1. Ensure that the logcheck.offset file has been created in the same directory as the log file being read.
  2. If the offset files are not being created, ensure that logtail is installed, and make sure the path in the logcheck.sh file is correct.
  3. Run logger to ensure that events are actually being logged.
  4. Run logcheck from the command line to confirm that mail will be sent to the correct user.
  5. Confirm that the cronjob for logcheck has been added.

Summary
Log file management is crucial to system security. In this Daily Drill Down, I discussed the swatch and logcheck programs, looked at the installation and configuration of each of these programs, and explained ways of getting the best results from these programs.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

Editor's Picks

Free Newsletters, In your Inbox