Software

Example B

Being able to interpret Linux log files is a tricky game of guess the syntax. Fortunately, there is some method to that madness, and Jim McIntyre is here to make it all seem simple.


The ability to interpret system log files provides administrators with an important tool for monitoring system activity. By reading and understanding the information contained in log files, administrators are able to determine if any unexpected activity has occurred. In this article, I will discuss the system logger utility, syslogd, and how to interpret information generated by the syslog utility.

How messages are logged
The default directory for system log files is /var/log. With most Linux distributions, the file /etc/syslog.conf is configured to write messages to at /var/log/messages. With Red Hat 6.0 and newer distributions, the /etc/syslog.conf file is preconfigured to write messages to the four files shown in Table A.

Table A
/var/log/secure The secure file is used to log any attempts to login as root and attempts to use the su command. This file also contains information on attempts to connect from remote systems and failed attempts to login as root.
/var/log/messages The messages file is the default log file for most systems. On some systems, this may be the only log file used. The messages file contains the following information: Any messages written to the console.
Operating system messages written to the kernel internal log buffer.
Any messages generated by programs using the syslog() system call. This includes sendmail, login, and named.
/var/log/maillog The maillog file contains reports on mail server status and records of incoming and outgoing mail.
/var/log/spooler The spooler file contains messages generated by activity on news servers and the uucp (UNIX-to-UNIX copy program). This file is unnecessary on systems not running these daemons.
The /etc/syslog.conf file is configured to write the above files.

Configuring syslog
On busy networks, there will be a lot of messages produced by syslog. Not all of these messages are important to the system administrator. To provide an easy way to read these messages, configure the /etc/syslog.conf file to reflect your own requirements. Before you create a system-logging configuration, you need to learn how syslogd categorizes and prioritizes log messages.

The syslogd daemon categorizes messages according to the activity or process that generates the message. These categories are referred to as facilities. Table B lists the facilities used by syslog.

Table B
Facility Category
syslog Messages generated by syslog
authpriv Private security and/or authorization
kern Messages generated by the kernel
user Messages generated by user-run programs
daemon Messages generated by system daemons
ftp Messages generated by FTP servers
mail Messages generated by mail servers
lpr Messages generated by the print daemon, lpr
news Messages generated by the network-news daemon, innd
auth/security Security and authorization
uucp Messages generated by uucp
cron Messages generated by the cron daemon
Log facilities used by syslog

To further categorize log messages, syslog is able to set priorities on messages. Table C lists the priorities used by syslog, in decreasing order of importance.

Table C
Priority Message generated
Emerg/panic The system has become unstable.
Alert This means a condition requires immediate attention.
Crit This signifies that a critical condition exists on the system.
Err/error These are error messages.
Warning/warn These are warning messages.
Notice This means there is an important condition requiring attention.
Info These are messages providing information on process and/or system status.
Debug These are messages used to debug errors.
None With this, no conditions are monitored.
Syslog priorities

The logging configuration is established by adding entries to /etc/syslog.conf. Each entry in the /etc/syslog.conf file must address:
  • The logging facility.
  • The priority.
  • The file where the log messages are written for the facility/priority combination.
The priority specified for any facility in /etc/syslog.conf automatically logs all messages generated at that priority and higher. For example, the following two entries in /etc/syslog.conf,authpriv.info    /var/log/secure
will log information generated by attempts to gain root privilege and connection attempts at the info level and higher to /var/log/messages. This means all messages generated at the notice, warning, error, crit, alert, and panic priority levels will be logged to /var/log/messages. By setting the initial priority too low for some facilities, the amount of logging information may become difficult to read. Monitor your log files carefully to ensure that you are able to distinguish the important information from the information generated by normal system use.
Typical configurations
Now, let's look at some typical configurations for the syslog.conf file.

The kernel is the heart of any Linux/UNIX system. Any messages generated by the kernel should immediately be made available to the administrator. To ensure that the administrator sees kernel messages immediately, use the following entry:
kern.*      /dev/console

The following entry will also write a copy of all kernel messages to /var/log/messages:
kern.*      /var/log/messages

Kernel and/or system panic messages should be sent to all users on your network, /var/log/messages, and the console. The following entry will ensure that this occurs:
*.emerg      *

All attempts at root access, successful or unsuccessful, should be logged in /var/log/secure. All user authorization attempts should be logged to /var/log/auth. All authorization attempts should be logged at the info level and higher. This is accomplished with the following two entries:
authpriv.info    /var/log/secure
auth.info     /var/log/auth


System daemons and the mail server will often use their own log files, making it unnecessary to log messages generated by these processes to /var/log/messages. Click here to view the entry for processes and daemons using their own logging system.
auth,authpriv,daemon,mail.none     /var/log/messages
The syslog does not create log files. The administrator must create these files. Log files are usually created using the touch command. For example, to create the /var/log/auth file, run the following command as root:touch /var/log/auth

Interpreting firewall log messages
All system administrators need to be able to interpret their firewall log files. Deciphering the information your firewall generates allows you to assess system security and identify system strengths and weaknesses.

The firewall logging process logs individually matched packets as kern.info messages for all firewall rules with the -l flag set. The default log file for firewall messages is /var/log/messages. Firewall log messages are also sent to the console.

It is good practice to log firewall messages to a second location, in addition to the default file, as in the following entry in /etc/syslog.conf:
kern.info /var/log/firewall
keeps a running log in the /var/log/firewall file.

Before we examine a firewall log message, let's take a look at a sample rule established using ipchains. This rule is used to deny access to port 111, which is the port mapped to the sunrpc and portmap services. Click here to view the sample rule.
ipchains -A input -p udp -I $EXTERNAL_INTERFACE -d $IPADDR 111 -j deny -l

When an attempt to access port 111 is made, a firewall log message similar to the one below may be generated. Because port 111 (portmap) is often targeted, this message is generated regularly. Click here to view the firewall log message.
Apr 3 22;16;23 sys-dev kernel: packet log: input DENY eth0 proto=17 192.168.10.21:23125 10.31.7.120:111 L=220 S=0x00 I=24560 F=0x0000 T=40

Now, let's examine the fields in this message. Table D shows the most important fields.

Table D
Field Meaning
Jun 9 This is for the date.
22;16;23 This field shows the time.
sys-dev This field shows the server hostname.
kernel: This indicates that the message is generated by the kernel log facility.
packet log This is appended to the facility name by IPFW.
input The rule is attached to the input firewall chain.
DENY This is the default action to take with this packet.
eth0 This is the network interface receiving or sending the packet.
proto=17 This is the protocol used by the packet.
192.168.10.21:23125 This is the IP address from which the packet originated.
 23125 This is the port from which the packet originated.
10.31.7.120 This is the destination IP address.
111 This is the destination port for which the packet was intended.
These are the most important fields in a firewall log.

The information listed above tells the administrator that on April 3, at 10:16 P.M., a UDP packet arrived at interface eth 0 from the source address 192.168.10.21. The packet originated from port 23125 on the source host. The packet was destined for port 111 on host sys-dev, with the assigned IP address 10.31.7.120. The packet was intended for port 111, which is the portmap port. Most importantly, the packet was rejected by ipchains.

Table E shows fields also generated with the log message.

Table E
Field Meaning
L=220 Packet length (220 bytes)
S=0x00 Type of service
I=243560 Packet ID, 243560
F=0x0000 Packet byte offset
T=52 Time-to-live (This is the maximum number of hops remaining before the packet is dropped.)
These are miscellaneous fields generated in firewall logs.

Common port scans
If you're unfamiliar with ports, the best analogy is to compare a port to a channel providing access to your system. In electronic eavesdropping, a common method to look for openings in a system is to use a scanner to listen to traffic on multiple frequencies (channels). When a vulnerable channel is found, I can either listen to the traffic on that channel or try to insert my own data and convince the other system that this data is legitimate.

Likewise, when I want to access another network, a good first step is to run a port scan on that system to gain some information on what ports are vulnerable to attack and to learn what services are running. This combination of open ports and available services is the starting point for the majority of attacks on networks.

Table F lists some of the ports used on most systems, the protocol used, and the vulnerability to attack.

Table F
Port Protocol Service Vulnerability Intention of attack
0 TCP/UDP reserved high No legitimate use
7 TCP/UDP echo high UDP attack
11 TCP SYSTAT high User processes
15 TCP NETSTAT high Network information, routing tables, open connections, etc.
21 / 20 TCP FTP high FTP services
23 TCP TELNET high TELNET services
25 TCP SMTP high Establish spam relay
53 TCP DOMAIN high DNS spoofing
109,110 TCP POP3 very high E-mail services
111 TCP/UDP SUNRPC /PORTMAP very high This port is exploited more than any other.
143 TCP IMAP very high IMAP services
520 UDP ROUTE high Routing tables
635 UDP MOUNTD high Use mountd daemon for access
1080 TCP SOCKS high Spam relay proxy server access
2049 TCP NFS high Compromise remotely mounted drives
These are some commonly scanned ports.

Fortunately, syslog is capable of informing you when an attempt to compromise your system has been made. The example below shows some log file entries generated when some common ports on Linux systems are scanned. This list is by no means exhaustive. These examples are meant to show you what the log entries for some typical port scans might look like. Some information has been intentionally omitted to keep the examples short; however, the most important information is included.

Click here to view Example A. In this example, a packet originating at IP address 192.168.10.31, port 21310 and destined for the host sys-dev, IP address 10.21.7.445, port 22, was denied access by ipchains.
Attempt to access port 23 for access through telnet
Apr 3  22;16;23 sys-dev daemon: packet log: input DENY eth0 PROTO=6 192.168.10.31:21310 10.21.7.45:23

Click here to view Example B. In this example, a packet destined for port 111 on the host sys-dev, IP address 10.21.7.45, was denied by ipchains at 6:23 P.M. on April 4.
Attempt to access port 111 (sunrpc/portmap)
Apr 4  18;23;45 sys-dev daemon: packet log: input DENY eth0 PROTO=6 192.168.10.31:21310 10.21.7.45:111

Making it easier to read system logs
The most efficient way to monitor your system log files is to have a utility or script monitor them either continuously or through a cron job. When these utilities or scripts detect a problem, an e-mail message is sent to the administrator. Here are two programs that are run on most systems and are easily configured to monitor log files.

Logcheck
Logcheck is used to periodically monitor your log files. When a suspect entry is found, the information is e-mailed to the administrator. Logcheck is one of the most common packages for monitoring logs and is available from the Psionic Web site.

Swatch
The simple watcher (swatch) program is a log file filter. When a pattern contained in the ruleset for swatch is matched, swatch performs one or more actions defined by the administrator. This program may be run continuously as a background process or at intervals as a cron job. The swatch package is available from most Linux vendors and most Web sites offering Linux security packages.

Summary
Regardless of the operating system in use on your network, there should be a way to log activity on the system. This activity includes both normal traffic generated by authorized users and attempts to infiltrate the network. Your logging system can provide you with a lot of information, but this information is only as good as your ability to read and understand it.

With this article, I have provided an introduction to reading and interpreting log files. I have covered the configuration of the syslog utility, syslog facilities and priorities, and how to add entries to the syslog.conf file to improve monitoring. I have also covered some common scans, focusing on the ports and protocols involved, and the type of access which may be gained from a successful attack on these ports. Finally, I have covered some popular programs to automate the process of monitoring and interpreting log files.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

Editor's Picks

Free Newsletters, In your Inbox