Linux

Log Linux services with runit

Vincent Danen explains how to configure logging for supervised services with the runit program.

In a previous tip, we looked at the runit program and how it can be used to supervise services, such as sshd. Each supervised service is controlled via a run script, similar to an init script for a system service. This run script does one thing: prepares for and starts a service.

The run script we used previously to start the OpenSSH sshd daemon was:

#!/bin/sh
exec /usr/sbin/sshd -D >/dev/null 2>&1

This starts sshd and passes it the -D option, which tells it not to detach and become a daemon. This is vital for supervised services as runsvdir can only directly control services which do not fork and place themselves in the background. Most services have similar options or debug-type options to prevent them from forking.

In the above, however, any output from sshd is sent to /dev/null, which means we could be missing some critical information from the service. Fortunately, runit also provides a program called svlogd, which takes the output of one service as input to another and writes it to a series of log files. So instead of having /var/service/sshd/run (which is the sshd run script) contain the snippet above, we would instead use:

#!/bin/sh
exec 2>&1
exec /usr/sbin/sshd -D

Now the output from sshd, both stdout and stderr, is not being redirected. Being a controlled service will not allow this information to be seen unless we create an associated log service to manage the log information. To do this, an additional directory needs to be created, called log/, and it needs to contain its own run script. In this sense, every supervised service can, in fact, have two services: the primary service itself (sshd), and the service's log service (sshd/log). To create the log service, create the /var/service/sshd/log directory and create /var/service/sshd/log/run with the following contents:

#!/bin/sh
cd /var/log/service/sshd
/sbin/chpst -u logger /sbin/svlogd -tt /var/log/service/sshd

This run script calls the program chpst, which comes with runit, and tells it to execute the next program as the user "logger" (you will need to create this user first). Otherwise, the program will be called as user root. It then tells it to write the output of the primary service to the logging directory, /var/log/service/sshd. This directory needs to be created and needs to have read, write, and execute permissions to the logger user.

svlogd is a smart program and will only create log files up to 1MB in size, by default. Once that size has been reached, it will automatically rename the "current" log file to one using a tai64n timestamp. The easiest way to describe this is to provide an illustration:

# ls /var/log/service/sshd/ -l
total 5220
-rw-r-----  1 logger logger 999015 Mar 31 06:14 @4000000047f0d5a2044617e4.s
-rw-r-----  1 logger logger 999053 Mar 31 06:56 @4000000047f0dfa52c9f4184.s
-rw-r-----  1 logger logger 999072 Mar 31 07:29 @4000000047f0e7282aa2aa4c.s
-rw-r-----  1 logger logger 999021 Mar 31 08:03 @4000000047f0ef3037db56dc.s
-rw-r-----  1 logger logger 999016 Mar 31 09:17 @4000000047f10079393fb93c.s
-rw-r-----  1 logger logger 322302 Mar 31 19:48 current
-rw-------  1 logger logger      0 Mar 29 21:15 lock

The "current" file is the current log file. The others are log files renamed to a file name using the precise timestamp notation (tai64n). By default, svlogd will keep a total of 10 log files, and will remove the oldest log file when rotating a new one. This will prevent any service from ever consuming more than 10MB (by default) of space for logs.

Using runit is fantastic; I've been using it for years and prefer nothing else. Supervised services is a great way for ensuring services are always running and, if they are taken down somehow by attack or accident, runsvdir will restart it within seconds. Nearly any service can be supervised, such as apache, sshd, and MySQL. With the reliability runit brings, there is no need for secondary "watch dog" programs to restart services if they die as runit can handle it all.

Get the PDF version of this tip.

Delivered each Tuesday, TechRepublic's free Linux and Open Source newsletter provides tips, articles, and other resources to help you hone your Linux skills. Automatically sign up today!

About

Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.

1 comments
TJ111
TJ111

Man, I wish I could setup runit on some systems, but I just won't have time for at least another 2 months, and you can't half ass these things. Bookmarking this and your other post in the hopes I can find it again in a month or two.

Editor's Picks