Jack Wallen shows you how to sync files across a network on Linux with a simple command line tool.
With Linux there are so many ways to synchronize and/or backup files over a network. For many, rsync and scp are the de facto standard. There is, of course, another option--one you've likely never heard of. That option is Unison, a free, open source, cross-platform bi-directional file sync tool. Unison is used to store two replicas that are modified separately and brought up-to-date by propagating changes to each store.
Unison is capable of synching directories on a local system or across a network. I want to show you how to use this tool and SSH to sync a directory on one Linux server to another. It's incredibly simple to use and even has a GUI that can also be installed, for those who prefer graphical tools over the command line. I'll be illustrating the command line version of Unison on two instances of Ubuntu Server.
SEE: Best Linux server distributions of 2020 (free PDF) (TechRepublic)
What you'll need
In order to work with Unison, you'll need two instances of Linux. As I mentioned, I'll be demonstrating with Ubuntu Server. This tool can also be installed on most every Linux distribution. The only difference in the installation is that, for CentOS, you'll have to enable the epel-release repository. You'll also need a user with sudo privileges (for the installation only).
How to install Unison
The installation of Unison is quite simple. You do have to install this software on both instances of Linux. So log in to them and issue the command:
sudo apt install unison -y
Make sure to run the above command on both Linux machines and you're ready to continue.
How to sync directories across a network
Let's say you have two directories:
~/data1 on the local drive
~/data2 on the remote drive
I'll first demonstrate how to sync those two files manually. After that, I'll show you how to make it a bit easier by doing so with a profile.
To sync across a network, Unison makes use of SSH. I would highly recommend setting up SSH key authorization for this. If you haven't already set up SSH key authentication, it can be done with two simple commands on the source server. Those commands are:
ssh-keygen ssh-copy-id SERVER
Where SERVER is the IP address of the destination server.
Once you have key authentication working, let's sync those two directories. To do this, issue the command on the source server:
unison ~/data1 ssh://USER@SERVER/data2
Where USER is the username on the remote machine and SERVER is the IP address of the remote machine.
The first time you use Unison on a machine, you'll see a message giving you a bit of information about the sync (Figure A).
Hit Return on your keyboard to continue with the sync. You will then be prompted to type f for each file to be synced. Once you've given the okay for each file, you'll then be prompted to type y to okay the sync (Figure B).
When the sync completes, you'll get your command prompt back.
Now, how do we make this a bit easier? We use the -batch option like so:
unison -batch ~/data1 ssh://USER@SERVER/data2
By using the -batch option, you'll not be prompted to answer any questions and your sync will complete.
How to use a Unison profile
We'll make this process even easier by making use of profiles. We'll sync the same directories to the same machines, but not have to type out the entire command. Create a new profile with the command:
You can name the profile anything you like, just leave the .prf extension intact.
A Unison profile looks like this:
root = /PATH root = ssh://USER@SERVER//PATH path = LOCAL_DIR path = REMOTE_DIR
Where PATH is the explicit path that houses the directories to be synced, USER is the remote username, SERVER is the IP address of the remote server, LOCAL_DIR is the local directory to be sync'd, and REMOTE_DIR is the remote directory to be synced.
Save and close that file.
To use the profile, the command would be:
If you want to include the -batch option in your profile, add the following to the end of the profile.prf file:
batch = true
How to automate this process
Unfortunately, the only way to automate this process is to password-less SSH key authentication and then create a cronjob to run every five minutes with a crontab entry like:
*/5 * * * * unison -batch ~/data1 ssh://USER@SERVER//home/USER/data2
Where USER is the remote username and SERVER is the remote server IP address.
Save the crontab file and, every five minutes, those two directories should be put in sync with one another.
Because this requires password-less authentication, this might not be the best option for syncing across your servers. If you're confident your LAN is secure, you might be okay with that option, but it's a pretty tough call to make--choose wisely.
That's how you can easily keep two directories, on Linux machines, in sync across your network with Unison. Get a bit creative and you can make this handy tool do even more.
How to become a developer: A cheat sheet (TechRepublic)
Kubernetes security guide (free PDF) (TechRepublic)
Linux file and directory management commands (TechRepublic Premium)
Linux, Android, and more open source tech coverage (TechRepublic on Flipboard)