The ability to interpret system log files provides administrators with an important tool for monitoring system activity. By reading and understanding the information contained in log files, administrators are able to determine if any unexpected activity has occurred. In this article, I will discuss the system logger utility, syslogd, and how to interpret information generated by the syslog utility.
How messages are logged
The default directory for system log files is /var/log. With most Linux distributions, the file /etc/syslog.conf is configured to write messages to at /var/log/messages. With Red Hat 6.0 and newer distributions, the /etc/syslog.conf file is preconfigured to write messages to the four files shown in Table A.
|/var/log/secure||The secure file is used to log any attempts to login as root and attempts to use the su command. This file also contains information on attempts to connect from remote systems and failed attempts to login as root.|
The messages file is the default log file for most systems. On some systems, this may be the only log file used. The messages file contains the following information:
Any messages written to the console.
Operating system messages written to the kernel internal log buffer.
Any messages generated by programs using the syslog() system call. This includes sendmail, login, and named.
|/var/log/maillog||The maillog file contains reports on mail server status and records of incoming and outgoing mail.|
|/var/log/spooler||The spooler file contains messages generated by activity on news servers and the uucp (UNIX-to-UNIX copy program). This file is unnecessary on systems not running these daemons.|
On busy networks, there will be a lot of messages produced by syslog. Not all of these messages are important to the system administrator. To provide an easy way to read these messages, configure the /etc/syslog.conf file to reflect your own requirements. Before you create a system-logging configuration, you need to learn how syslogd categorizes and prioritizes log messages.
The syslogd daemon categorizes messages according to the activity or process that generates the message. These categories are referred to as facilities. Table B lists the facilities used by syslog.
|syslog||Messages generated by syslog|
|authpriv||Private security and/or authorization|
|kern||Messages generated by the kernel|
|user||Messages generated by user-run programs|
|daemon||Messages generated by system daemons|
|ftp||Messages generated by FTP servers|
|Messages generated by mail servers|
|lpr||Messages generated by the print daemon, lpr|
|news||Messages generated by the network-news daemon, innd|
|auth/security||Security and authorization|
|uucp||Messages generated by uucp|
|cron||Messages generated by the cron daemon|
To further categorize log messages, syslog is able to set priorities on messages. Table C lists the priorities used by syslog, in decreasing order of importance.
|Emerg/panic||The system has become unstable.|
|Alert||This means a condition requires immediate attention.|
|Crit||This signifies that a critical condition exists on the system.|
|Err/error||These are error messages.|
|Warning/warn||These are warning messages.|
|Notice||This means there is an important condition requiring attention.|
|Info||These are messages providing information on process and/or system status.|
|Debug||These are messages used to debug errors.|
|None||With this, no conditions are monitored.|
The logging configuration is established by adding entries to /etc/syslog.conf. Each entry in the /etc/syslog.conf file must address:
- The logging facility.
- The priority.
- The file where the log messages are written for the facility/priority combination.
will log information generated by attempts to gain root privilege and connection attempts at the info level and higher to /var/log/messages. This means all messages generated at the notice, warning, error, crit, alert, and panic priority levels will be logged to /var/log/messages. By setting the initial priority too low for some facilities, the amount of logging information may become difficult to read. Monitor your log files carefully to ensure that you are able to distinguish the important information from the information generated by normal system use.
Now, let's look at some typical configurations for the syslog.conf file.
The kernel is the heart of any Linux/UNIX system. Any messages generated by the kernel should immediately be made available to the administrator. To ensure that the administrator sees kernel messages immediately, use the following entry:
The following entry will also write a copy of all kernel messages to /var/log/messages:
Kernel and/or system panic messages should be sent to all users on your network, /var/log/messages, and the console. The following entry will ensure that this occurs:
All attempts at root access, successful or unsuccessful, should be logged in /var/log/secure. All user authorization attempts should be logged to /var/log/auth. All authorization attempts should be logged at the info level and higher. This is accomplished with the following two entries:
System daemons and the mail server will often use their own log files, making it unnecessary to log messages generated by these processes to /var/log/messages. Click here to view the entry for processes and daemons using their own logging system.
auth,authpriv,daemon,mail.none /var/log/messagesThe syslog does not create log files. The administrator must create these files. Log files are usually created using the touch command. For example, to create the /var/log/auth file, run the following command as root:touch /var/log/auth
Interpreting firewall log messages
All system administrators need to be able to interpret their firewall log files. Deciphering the information your firewall generates allows you to assess system security and identify system strengths and weaknesses.
The firewall logging process logs individually matched packets as kern.info messages for all firewall rules with the -l flag set. The default log file for firewall messages is /var/log/messages. Firewall log messages are also sent to the console.
It is good practice to log firewall messages to a second location, in addition to the default file, as in the following entry in /etc/syslog.conf:
keeps a running log in the /var/log/firewall file.
Before we examine a firewall log message, let's take a look at a sample rule established using ipchains. This rule is used to deny access to port 111, which is the port mapped to the sunrpc and portmap services. Click here to view the sample rule.
ipchains -A input -p udp -I $EXTERNAL_INTERFACE -d $IPADDR 111 -j deny -l
When an attempt to access port 111 is made, a firewall log message similar to the one below may be generated. Because port 111 (portmap) is often targeted, this message is generated regularly. Click here to view the firewall log message.
Apr 3 22;16;23 sys-dev kernel: packet log: input DENY eth0 proto=17 192.168.10.21:23125 10.31.7.120:111 L=220 S=0x00 I=24560 F=0x0000 T=40
Now, let's examine the fields in this message. Table D shows the most important fields.
|Jun 9||This is for the date.|
|22;16;23||This field shows the time.|
|sys-dev||This field shows the server hostname.|
|kernel:||This indicates that the message is generated by the kernel log facility.|
|packet log||This is appended to the facility name by IPFW.|
|input||The rule is attached to the input firewall chain.|
|DENY||This is the default action to take with this packet.|
|eth0||This is the network interface receiving or sending the packet.|
|proto=17||This is the protocol used by the packet.|
|192.168.10.21:23125||This is the IP address from which the packet originated.|
|23125||This is the port from which the packet originated.|
|10.31.7.120||This is the destination IP address.|
|111||This is the destination port for which the packet was intended.|
The information listed above tells the administrator that on April 3, at 10:16 P.M., a UDP packet arrived at interface eth 0 from the source address 192.168.10.21. The packet originated from port 23125 on the source host. The packet was destined for port 111 on host sys-dev, with the assigned IP address 10.31.7.120. The packet was intended for port 111, which is the portmap port. Most importantly, the packet was rejected by ipchains.
Table E shows fields also generated with the log message.
|L=220||Packet length (220 bytes)|
|S=0x00||Type of service|
|I=243560||Packet ID, 243560|
|F=0x0000||Packet byte offset|
|T=52||Time-to-live (This is the maximum number of hops remaining before the packet is dropped.)|
Common port scans
If you're unfamiliar with ports, the best analogy is to compare a port to a channel providing access to your system. In electronic eavesdropping, a common method to look for openings in a system is to use a scanner to listen to traffic on multiple frequencies (channels). When a vulnerable channel is found, I can either listen to the traffic on that channel or try to insert my own data and convince the other system that this data is legitimate.
Likewise, when I want to access another network, a good first step is to run a port scan on that system to gain some information on what ports are vulnerable to attack and to learn what services are running. This combination of open ports and available services is the starting point for the majority of attacks on networks.
Table F lists some of the ports used on most systems, the protocol used, and the vulnerability to attack.
|Port||Protocol||Service||Vulnerability||Intention of attack|
|0||TCP/UDP||reserved||high||No legitimate use|
|15||TCP||NETSTAT||high||Network information, routing tables, open connections, etc.|
|21 / 20||TCP||FTP||high||FTP services|
|25||TCP||SMTP||high||Establish spam relay|
|109,110||TCP||POP3||very high||E-mail services|
|111||TCP/UDP||SUNRPC /PORTMAP||very high||This port is exploited more than any other.|
|143||TCP||IMAP||very high||IMAP services|
|635||UDP||MOUNTD||high||Use mountd daemon for access|
|1080||TCP||SOCKS||high||Spam relay proxy server access|
|2049||TCP||NFS||high||Compromise remotely mounted drives|
Fortunately, syslog is capable of informing you when an attempt to compromise your system has been made. The example below shows some log file entries generated when some common ports on Linux systems are scanned. This list is by no means exhaustive. These examples are meant to show you what the log entries for some typical port scans might look like. Some information has been intentionally omitted to keep the examples short; however, the most important information is included.
Click here to view Example A. In this example, a packet originating at IP address 192.168.10.31, port 21310 and destined for the host sys-dev, IP address 10.21.7.445, port 22, was denied access by ipchains.
Attempt to access port 23 for access through telnet
Apr 3 22;16;23 sys-dev daemon: packet log: input DENY eth0 PROTO=6 192.168.10.31:21310 10.21.7.45:23
Click here to view Example B. In this example, a packet destined for port 111 on the host sys-dev, IP address 10.21.7.45, was denied by ipchains at 6:23 P.M. on April 4.
Attempt to access port 111 (sunrpc/portmap)
Apr 4 18;23;45 sys-dev daemon: packet log: input DENY eth0 PROTO=6 192.168.10.31:21310 10.21.7.45:111
Making it easier to read system logs
The most efficient way to monitor your system log files is to have a utility or script monitor them either continuously or through a cron job. When these utilities or scripts detect a problem, an e-mail message is sent to the administrator. Here are two programs that are run on most systems and are easily configured to monitor log files.
Logcheck is used to periodically monitor your log files. When a suspect entry is found, the information is e-mailed to the administrator. Logcheck is one of the most common packages for monitoring logs and is available from the Psionic Web site.
The simple watcher (swatch) program is a log file filter. When a pattern contained in the ruleset for swatch is matched, swatch performs one or more actions defined by the administrator. This program may be run continuously as a background process or at intervals as a cron job. The swatch package is available from most Linux vendors and most Web sites offering Linux security packages.
Regardless of the operating system in use on your network, there should be a way to log activity on the system. This activity includes both normal traffic generated by authorized users and attempts to infiltrate the network. Your logging system can provide you with a lot of information, but this information is only as good as your ability to read and understand it.
With this article, I have provided an introduction to reading and interpreting log files. I have covered the configuration of the syslog utility, syslog facilities and priorities, and how to add entries to the syslog.conf file to improve monitoring. I have also covered some common scans, focusing on the ports and protocols involved, and the type of access which may be gained from a successful attack on these ports. Finally, I have covered some popular programs to automate the process of monitoring and interpreting log files.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.