As industry trends have shown, Linux is taking server market share from both Microsoft Windows and especially proprietary UNIX systems to become the number three server operating system. Linux is rapidly becoming the major contender Microsoft must face in the low-end server market. One of the reasons for this is security concerns, and another is cost. Whether you are implementing automated backup and security policies for anything from a single workstation to a network of servers and workstations, the rsync utility can address both contingencies, especially when properly used in conjunction with other tools to enhance security. This article focuses on how admins can use rsync to secure their servers.
The rsync utility is, by design, a simple file copying tool. It has an advantage over utilities like cp and rcp, however, in that it uses a checksum comparison to determine what actually needs to be copied and what is already at the destination. For instance, if you use rsync to copy a directory with thirty files in it from computer foo to computer bar, cp will just start copying files, and will overwrite files with the same name on bar as the files being copied from foo. On the other hand, after the first time you use rsync to copy files from foo to bar, rsync will do a quick comparison of the same directory on foo and bar to see what is different, and will only copy the files that differ between the two. This can greatly reduce the amount of time spent on file system replication, such as when backing up data on one computer to a hard drive on another computer.
The rsync utility is very straightforward and simple to use, but it must be used wisely to ensure proper security. It can also be used as a security tool in and of itself, without necessarily using it for anything else, to do file system integrity auditing.
Secure file copying
Data security requires backups of your critical data. While this can be accomplished the way many mom-and-pop businesses still do, by going to each workstation in the office and burning important files to CD-R, it is better in a networked office to have some kind of automated backup system running. The most efficient way to do this these days with the low cost and high speed of hard drives is often with a backup server, where files are copied over the network to hard drives attached to the backup server on a regular schedule. A computer old enough to be considered obsolete for many office tasks can serve admirably in such a simple role, especially if backups are performed in the middle of the night or on weekends when nobody is using the network.
Copying files between computers over the network with rsync is a simple procedure. At its simplest, you can enter the following command from a Linux system that contains the files you want to copy:rsync localdir host:remotedir
This invokes the rsync command with only default behaviors. It specifies the directory on the local system (localdir) from which you'll be copying, and specifies the hostname of the remote system (host) and the directory on that machine (remotedir) to which you'll be copying files. For instance, if you copy the directory /home/baz from host foo to directory /home/backup/qux on host bar, your command on host foo looks like this:
The operation ends up being performed entirely "in the clear", however, without much in the way of security, if that is all you do. It also doesn't copy symbolic links, or preserve permissions, user and group ownership, or creation times. For these, you'd add the -a option (for "archive") and the -e option to specify a remote shell program to use for security. That remote shell should use an encrypted protocol such as SSH for security purposes so that the remote shell option takes the form -e ssh. Your command, tunneling rsync through ssh, looks like this:rsync –a –e ssh localdir host:remotedir
If you want to get a better look at what is going on when you copy files across, you can use the -v and —progress options. The -v option specifies verbose output. The default behavior of rsync is to run silently, simply performing its task without bothering the user. Used once, the -v option tells you what files are being copied and gives you a summary at the end. Used twice, by adding -vv to the command, causes rsync to also tell you what files are being excluded and gives you more information in the final summary.
The —progress option, meanwhile, will give you an ongoing indicator of progress including the current file size, the percentage of the total transfer that has been completed, the current speed of transfer, and the estimated time remaining before the transfer is done. A command using double-verbosity with archive mode and ssh tunneling, and requiring a progress indicator, takes this form:rsync –avv –e ssh –progress localdir host:remotedir
If a transfer is interrupted in the middle of copying a file, rsync's normal behavior is to delete the partially copied file from the destination. You can use the —partial option to prevent rsync from deleting partially-copied files if you know in advance that you will be attempting to copy the file again in the event of an interrupted transfer. This speeds up the transfer on the second attempt, as rsync only copies data not already copied. If you are using both —partial and —progress, rsync provides a shorthand option that can be used for both; the -P option. Thus, to do a double-verbose archive copy with deletion of partial files turned off, requiring a progress indicator, and tunneling through ssh, your command takes the following form:rsync –avvP –e ssh localdir host:remotedir
To automate the process according to a schedule, you can simply add the rsync command to a cron job. You can accomplish this by adding an entry to the /etc/crontab file, like so:30 4 * * * user rsync –aP -e ssh localdir host:remotedir
The first five fields in a cron job entry define when the job is scheduled, and the sixth tells crontab what user will run the job. Replace the "user" string in the example with the name of the user account that will be running the job. The syntax of this cron job will cause it to be run once per day, at 4:30 in the morning.
The order of the first five fields is minutes, hours, day of the month, month of the year, and day of the week. The time fields start at 0, so that minutes are 0-59, and hours are 0-23. The next two fields, for days of the month and months of the year, are 1-31 and 1-12. The last field, day of the week, is 0-7, where either 0 or 7 indicate Sunday. The configuration of cron jobs can become much more complex than this, but this is the basics of creating a cron job to schedule data backups with rsync. For more information about how to create cron jobs, start with the commands man cron, man crontab, and man 5 crontab.
If you want to centrally manage your data backup cron jobs, you can create crontab entries on the backup server like this:30 4 * * * user rsync –aP –e ssh host:remotedir localdir
Set up that way, rsync will grab files from the remote directory (remotedir) on the specified remote host and copy them to the local directory you indicate (localdir).
Normally, filesystem integrity auditing is an operation performed by way of dedicated security tools like tripwire. Such tools tend to be very thorough and provide a lot of power and flexibility in how the sysadmin goes about performing filesystem integrity audits. There may be any number of reasons for a sysadmin to prefer a different means of filesystem integrity auditing, however.
Perhaps surprisingly, the rsync utility can be used to do integrity audits. Its checksum comparisons between primary and archived copies of files and filesystems can be used to determine what changes have been made to a filesystem, and those changes can then be examined to determine whether or not they were authorized changes.
The first thing you need to do in the process of using rsync for filesystem integrity auditing is to create a snapshot of your filesystem. This must be accomplished before there is need to check your filesystem's integrity, because you must have a "known good" copy against which to check the filesystem.
To do this, simply run an rsync copy command from the command line. To be smart about it, though—and to ensure that you are using a version of rsync that cannot have itself been compromised by an intruder, so that it won't detect or report changes to the filesystem—you should copy the rsync binary from the computer (probably at /usr/bin/rsync) to a non-rewritable medium, such as a CD-R. Then, when you specify the rsync utility to use, you should indicate the rsync binary on the mounted CD-R instead of the one on your computer. For instance, running this command copies files from one computer to another, if your CD is mounted at /mnt/cdrom:rsync –av —rsync-path=/mnt/cdrom/rsync –e ssh host:/ localdir
The above copies the entire filesystem from a remote host (the computer whose filesystem integrity you want to check) to a local directory on the machine you are using to run rsync. Once you have that snapshot of your filesystem, you can check filesystem integrity using the following command:rsync –avn —rsync-path=/mnt/cdrom/rsync –e ssh host:/ localdir
The -n option in the above command ensures that nothing is copied. Instead, rsync merely reports the differences between the working copy and the archive copy. If unexpected differences are shown, you should examine them to ensure that they are not unauthorized changes. If there are changes, a list of changed files will be displayed.
Changes to utilities such as /bin/ls, /bin/bash, and /bin/sshd are highly suspect, and if no such changes should have been made, you should consider your system compromised. If you have an automated system update running, however, those files may have been legitimately changed by that process. As such, it is a good idea to have automated system updates log any updated packages so you can compare those changes with the list of changes your rsync integrity audit reports.
Bringing it together
Your automated backups can be combined with your integrity auditing. For instance, you may use automated backups to back up the entire system, and create a similar cron job that occurs before the backup to run an integrity audit, with a redirect to write its output to a logfile. Under such circumstances, it is usually a good idea to have your audit only check parts of the filesystem that should not change, while having your backups target the entire filesystem.
For instance, your backups might target the root directory while your integrity audits omit directories whose contents regularly change such as /home and /var. To be too zealous in deciding what will be checked by an automated filesystem audit is likely to result in the automation of backups becoming entirely ineffective.
If you really want to get fancy, you can even write a shell script that is run by your cron job, wherein it does integrity auditing first, then runs the backup only if there are no problems with the integrity audit. Then, when you check the log of the integrity audit for yourself later, if there's nothing wrong you can run the backup at your leisure to get up to date (preferably after another integrity audit, to ensure nothing has changed since the automated audit). Clearly, shell scripting is beyond the scope of this article, but the ability to write shell or Perl scripts is a handy skill for a Linux sysadmin to have.
Automated backups are reasonably simple with rsync, which is to be expected since backups are what rsync was designed to do. However, the way rsync can be used to perform other tasks as well, such as filesystem integrity auditing, illustrates the flexibility Linux provides by the development of many small utilities that each do one thing well, as opposed to large programs that each try to do everything. Such small utilities can then be combined to perform more complex tasks, as demonstrated by the ability to tunnel rsync through SSH to encrypt and compress data transfers over the network, to greatly enhance data and system security simply and easily by use of simple command line utilities. In the hands of a knowledgeable system administrator, rsync becomes a versatile and effective security tool.
Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.