Configure IT Quick: Log Linux system events with syslog

Get a step-by-step look at how to install, use, and optimize syslog for centralized login on Linux.

A system administrator must have the ability to track activity on the network. Linux provides this capability through the syslog package. This Daily Drill Down will provide a step-by-step process for installing and configuring the syslog and klogd daemons, illustrate the advantages and disadvantages of various configurations, and show how to secure system log files. We’ll also look at ways to optimize system-logging performance. Although there are two separate daemons associated with the sysklog package, this Daily Drill Down will refer to both daemons collectively as syslog.

What is syslog?
Syslog is a daemon that acts as an interface between the log files on a Linux system and any program that’s required to generate log messages. When a program sends a record of an event to syslog, the syslog daemon reads the /etc/syslog.conf file to determine the rules for logging messages from the application. Syslog will then perform one or more of the following actions with the message:
  • The message may be written to a file specified in /etc/syslog.conf.
  • The message may be mailed to a user or sent to the user's terminal.
  • The message may be sent to another system.
  • The message may be forwarded to another program.

The action taken upon receipt of a message is determined by:
  • The facility sending the message.
  • The level of importance assigned to the message.
  • The action specified.

The facility, level, and the action taken are all specified in the /etc/syslog.conf file.

The /etc/syslog.conf file
The /etc/syslog.conf file establishes rules for the behavior of the syslog daemon. The /etc/syslog.conf file is a text file that accepts entries in the format
facility.level       <tab>  action

An example of this format would be:     /var/log/maillog
Blank lines are ignored, and lines beginning with a pound sign (#) are comments.
The term facility is simply another way of describing activity resulting from system use. Each daemon or utility that generates syslog messages will use one of the facilities listed in Table A; for example, the lpr daemon uses the lpr facility. Messages generated by users use the use facility, and messages generated by daemons use the daemon facility. Multiple facilities, and multiple actions, may be assigned to the same link by separating each facility with a comma (,). For example, the following line
lpr,user.warning     root, jim

sends all warning-level (or higher) messages to the root user and to the user jim. Table A lists the facilities used by syslog.

syslog facilities
auth Authentication-related activity, e.g., pam_pwdb. auth has been deprecated (made redundant) by authpriv but may still be used.
authpriv Authentication activity that may contain secure information, e.g., usernames.
cron Activity associated with the cron and at daemons.
daemon Activity associated with daemons. Examples would be finger and FTP.
kern Kernel-related activity. These messages are normally written to a log file by klogd.
lpr Activity related to printing.
mail E-mail-related activity.
mark A syslog facility used to generate timestamps in log files.
news Activity related to the Internet News Server.
syslog Messages produced by syslog.
user Messages generated by user programs.
uucp Messages generated by the UNIX-to-UNIX Copy Protocol.
local10-local17 Facilities used with customized programs.
* Wildcard. Used for all facilities except mark.
Table A

The level specifies the minimum level of importance a message must have before it is logged. When messages with several levels of importance are required to be logged, select the minimum level you require. All messages at this level or higher will automatically be logged. For example, if the /etc/syslog.conf file contains the following line
inetd,kern,crit /var/log/critical

all messages generated by either the inetd daemon or the kernel with level of critical or higher will be sent to the /var/log/critical file.

Table B lists the levels used by syslog from highest to lowest priority.

syslog levels
Level Function
emerg or panic emerg is used when the system is unstable. panic is now deprecated.
alert A condition needs the immediate attention of the system administrator.
crit An error condition has occurred, and another daemon is adversely affected.
err An error condition affecting the functionality of another daemon or utility.
warning A warning message.
notice A normal condition that is significant to system performance.
info An information message.
debug Messages specific to programs or daemons that do not reflect problems.
none No level. Used to exclude a facility when wildcards are used.
Table B

Multiple selectors may also be specified on the same line. The line

configures syslog to log all facilities at level warning and above, except for inetd and user. For user, no logging is being used.

The actions filed in /etc/syslog.conf specify what syslog does when a message is received. These actions may include using messages in ways that are specifically performed by syslog. An example of this would be when an e-mail message is generated upon receipt of a message.

Table C lists the actions syslog may perform when it receives a message.

Actions performed by syslog
file The fully qualified path to a specific log file, e.g., /var/log/maillog.
terminal / printer The name of a system device, e.g., /dev/lp1 or /dev/console.
@hostname A remote hostname preceded by the @ symbol. The remote host must be running syslogd and be configured to receive information from a specified incoming selector.
username Specifies a user to receive syslog messages, via the write utility.
named pipe A fully qualified path to a specified FIFO (first in, first out) file. The named pipe causes syslog to write the message to the specified file.
Table C

Configuring syslog
The system-logging process is configured through the /etc/syslog.conf file. There are three questions to answer when configuring syslog.conf:
  1. What messages need to be logged?
  2. How should syslog process the messages?
  3. How soon do you need to receive the messages?

The only way to answer these questions is to know your network. After you’ve become familiar with your system and the messages it generates, you’ll be able to anticipate the logging requirements for the system.

Example A lists the default /etc/syslog.conf file from a Red Hat 6 server.

Example A: Default /etc/syslog.conf file1. # Log all kernel messages to the console.
2. # Logging much else clutters up the screen.

3. kern.*                                /dev/console
4. # Log anything (except mail) of level info or higher.
5. # Don't log private authentication messages!
6. *.info;mail.none;authpriv.none        /var/log/messages

7. # The authpriv file has restricted access.
8. authpriv.*                            /var/log/secure

9. # Log all the mail messages in one place.
10. mail.*                               /var/log/maillog

11. # Everybody gets emergency messages, plus log them on another
12. # machine.
13. *.emerg                              *

14. # Save mail and news errors of level err and higher in a
15. # special file.
16. uucp,lpr,news.crit                   /var/log/spooler

17. # Save boot messages also to boot.log
18. #local7.*                            /var/log/boot.log

Line numbers have been added to the above file for clarity.
The third line
#kern.* /dev/console,

is commented out (by removing the #) so all kernel messages will be logged to the console.

The sixth line
*.info;mail.none;authpriv.none /var/log/messages

specifies that all messages with a level of info or higher, except for mail and authpriv messages, will be logged to /var/log/messages. This line will also cause kernel messages to be written to /var/log/messages. If the third line is uncommented, an identical kernel message will be sent to the console.

Line 8
authpriv.* /var/log/secure

specifies that all authpriv messages are logged in /var/log/secure. The permissions on this file give read-write permission to root only.

Line 10
mail.* /var/log/maillog

specifies that all mail-related messages are logged to /var/log/maillog. Mail-related messages should be logged separately from other messages. This is very important on systems being used as mail servers.

Line 13
*.emerg *

specifies that all emergency messages are sent to all logged-in users. This is accomplished through the write utility.

Line 16
uucp,lpr,news.crit /var/log/spooler

specifies that all uucp, printing, and news messages at the crit level or higher are written to /var/log/spooler.
The lpr entry is included in this line for convenience.
Restarting syslog
After changes are made to /etc/syslog.conf, the syslog daemon itself must be restarted so that it can use the new configuration specified in the syslog.conf file. This is accomplished with the command
kill -HUP syslogd

Log file structures
There are two ways for the administrator to build log files. These are:
  1. Have most or all messages written to one log file. This file may then be processed for information by another utility, e.g., swatch or logcheck.
  2. Build log files based on functionality. This may include a separate log file for kernel messages, another for authentication messages, another for FTP file transfer records, etc.

Building log files based on functionality makes log-file management a much simpler process. If messages relating to authentication are required for analysis, the administrator is not required to sort through messages related to other services, such as mail or kernel messages. This approach also makes it easier to write shell scripts or programs to process syslog information.

Using a central log server
Using a centralized log server provides two benefits:
  1. Ease of administration
  2. A more secure logging system

The most important aspect of increasing security is to make sure that the server itself is secure. A busy FTP server or an e-mail server is not a good choice for your log server. Select a system that doesn't get a lot of login activity and doesn't provide a lot of Internet-related services.

Configuring syslog for a centralized server
There are two tasks to complete to configure your system to write messages to a centralized log server:
  1. Configure the syslog utility on each client to write messages to the centralized server.
  2. Implement time synchronization so that all hosts agree on the time of day.

Configuring syslog for a central log server
To configure the client systems to write log messages to the log server log-admin, configure the /etc/syslog.conf file on each system. Example B shows the default syslog.conf file configured for this purpose.

Example B: /etc/syslog.conf configured for central logging kern.*        /dev/console
 *.info;mail.none;authpriv.none  /var/log/messages
 authpriv.*       /var/log/secure
 mail.*       /var/log/maillog
 *.emerg        *
 uucp,lpr,news.crit     /var/log/spooler
 *.info        @log-admin

The line
*.info @log-admin

will configure syslog on the client system to log all messages to the selector *.info on both the client and the log-admin log server. This line will not be placed into the /etc/syslog.conf file on the log-admin server.

To enable the log server to accept log messages from hosts on the network, open the /etc/rc.d/init.d/syslog script and find the following section:
    echo -n "Starting system logger: "
    # we don't want the MARK ticks
    daemon syslogd -m 0
    echo -n "Starting kernel logger: "
    daemon klogd
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/syslog

Change the line
daemon syslogd -m 0

daemon syslogd -r

This will enable syslogd to receive log messages from network clients and will establish a default timestamp interval of 20 minutes. Timestamps are useful when log files are analyzed. If events occur regularly on the system, timestamps can be used to establish a pattern for these events. For example, if a server crashes, timestamps will tell you that the crash occurred between 4:00 A.M. and 4:20 A.M. and not just "sometime last night."

The next step in configuring a centralized log server is to synchronize the system clocks on the network. This is easily accomplished using the rdate command. As root, enter the following command on each host using admin-log for logging:
rdate -ps admin.log

This command will print the time received from the remote system and will set the local system to the same time as the remote system.

Two other utilities are available for controlling a system:
  • timed: the time daemon
  • xntp: the network time protocol

The klogd daemon
The klogd daemon logs kernel messages. The default for klogd is to send all messages to syslog. However, if klogd is started with the command
klogd -f <filename>

all messages will be written to the specified file, not syslog. The /etc/syslog.conf file is used as the configuration file for klogd because klogd does not have its own configuration file. An inexperienced administrator may use klogd to receive messages when the kernel becomes unstable. A more experienced administrator who is familiar with the kernel source code may use klogd to determine if the kernel on a system has been hacked.

In this Daily Drill Down, I discussed the processes involved in logging events on Linux systems. I also discussed the system-logging daemon, syslog, and the kernel-logging daemon, klogd. I demonstrated the types of messages generated by a Linux system and actions syslogd may perform when these messages are generated. Finally, I described the steps required to configure a centralized log server and showed how to edit the /etc/syslog.conf configuration file for syslogd.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

Editor's Picks