Data Management

Use rpm to automate customized network-wide upgrades

Automate your Linux server farm upgrades with this simple, reliable, and cost-effective solution from Jack Wallen, Jr.


When it comes to network-wide desktop or server upgrades, automation is the only real option. Without it, you’d be taking the long and difficult way around what should be a walk in the park.

The trick is finding a way to automate mass upgrades that is flexible; dependable; and, once it’s up-and-running, fast. Fortunately, rpm is here to help. The process is complex, but the end result is well worth the effort. In short, what you’ll need to do is set up a centralized location in which to place your rpm files (the files that will be used for upgrading). Create a new directory called rpmupdates and locate it in the document root of your Web server (in Red Hat 7.x, you would place this directory in /var/www/html/), connect to that centralized Web server via the network file system (NFS), and install into root's crontab a command that will automate the upgrade process.

In this Daily Drill Down, I will go through this multipart process so that you too can reap the rewards of customized and automated Linux server updates.

Why so complex?
This automated update system is incredibly complex due to a major limitation of the rpm tool: It can’t remotely upgrade or install multiple files. For remote installation or upgrades, rpm must have the exact filename of the package.

In order to get around that problem, rpm requires the help of a utility such as NFS so that multiple files will be seen as “local.”

On a local machine, you could use rpm to do a mass update. For example, you could have one directory, named /rpmudates, to house all of your new rpm files. As root, you could run rpm -Fvh /rpmupdates/*.rpm, and each rpm file that is newer than the file currently installed on the computer would be updated.

If you were able to update with rpm in this manner from a centrally located server, it should just be a matter of inserting the command:
rpm -Fvh http://IP_OF_SERVER/rpmupdates/*.rpm

(where IP_OF_SERVER is the IP address of your centralized server and assuming that the directory /var/www/html/rpmupdates houses your rpm files).

Unfortunately, this can’t be done because, as I stated earlier, updating remotely with rpm can only succeed with an explicit filename. This means that you would have to run rpm on every filename housed in your centralized server, which would be impractical, to say the least.

So in order to make this work, you need to allow your servers to connect via NFS. Because of this, the command will think that the /rpmupdates directory is housed locally and, therefore, will run successfully.

NFS
To get NFS working, you are going to have to have root access on each server, and you will have to edit some pretty critical files. Be very careful from here on out. You should also note that it's well known that NFS has a number of security issues. If this system is going to be contained within a private LAN, however, you shouldn't have to worry about security (beyond the trustworthiness of employees with LAN access).

You will need to make sure that you have user accounts that are identical on each machine, as well. Since much of this will be done as root, because you are upgrading rpm files, ensuring that there are identical user accounts should not be a problem, so long as the root passwords match.

Finally, you will assume that the IP addresses of the setup are as listed in Table 1.

Table 1
Address Role
192.168.1.1 Your centralized server
192.168.1.2 Server1
192.168.1.3 Server2
192.168.1.4 Server3
192.168.1.5 Server4
192.168.1.6 Server5

The server
The first thing you will deal with is the centralized server. On this system, you will modify /etc/hosts.deny and /etc/hosts.allow so that they will accept portmap requests from the client machines. On this same machine, you will enable the proper daemons and modify /etc/exports so the system will know exactly what to share out.

Before you actually begin setting up this service on the server, make sure you create a directory to share out to the clients. In a perfect world, you would make this a separate partition (or drive) altogether. If you don't live in a perfect world, you can create a directory called /rpmupdates. You will have to create this directory as the root users.

With that directory in place, you are ready to roll. The first thing you will do is fire up a console window (more than likely, you will be dealing with an X'less Linux, since this will be a server) and cd to the /etc directory. Once in /etc (pronounced etsee), fire up your favorite text editor (mine is Pico), open up the /etc/hosts.deny file, and append the following to the bottom of the file:
portmap: ALL

Save that file, and then open up /etc/hosts.allow. Here you will add the following (your IP addresses and Netmasks will differ depending on your setup):
portmap: 192.168.1.2/255.255.255.0
portmap: 192.168.1.3/255.255.255.0
portmap: 192.168.1.4/255.255.255.0
portmap: 192.168.1.5/255.255.255.0


Wildcards
You could use a wildcard in this situation. Say, for example, only your server farm uses the 192.168.1.x addressing scheme within your company network. If this is the case, you could enter portmap: 192.168.1.*/255.255.255.0 in the /etc/hosts.allow file and be done with it. This would also make expansion much easier.

The next step on the server is to get your daemons up and running. The daemons you want running are: portmap, rpcmountd, and nfsd.

Before you configure the daemons, check to see if they are working properly on the system. To do this, run the following commands:
/etc/rc.d/init.d/portmap start
/etc/rc.d/init.d/nfs start

Now the daemons should be accepting calls. You can check that they are with the following commands:
/etc/rc.d/init.d/portmap status
/etc/rc.d/init.d/nfs status

You should see that each service is running by issuing the above commands.

These daemons can be started manually or they can be started at boot. You will want to set these services up to start at boot. You can do this by issuing the /usr/sbin/setup command. From here, you want to scroll down to the System Services option and, using the space bar, select netfs, nfs, nfslock, portmap, rstatd, and ruserd. After you've selected these services, tab down to OK and then tab twice to quit.

Once the daemons are all running, you will want to configure the /rpmupdates directory to be shared out. To do this, open up the /etc/exports and add the following line (I'll use the wildcard this time):
/rpmupdates 192.168.1.* (ro)

Save the /etc/exports file and finally, as root, run the command:
exportfs -rv

and the additions to the /etc/exports file will be added to the /var/lib/nfs/xtab database.

The clients (er...servers)
Now you are going to configure our “clients” to connect to the centralized server. To do this, you will edit the /etc/fstab file. Do this with caution! Breaking this file means breaking your server's ability to boot! Before you actually add the new entry to the /etc/fstab file, it would be best to check to make sure your clients can connect to the server. Before you run this check, you will have to create a mounting directory on each client. For simplicity’s sake, keep this consistent. To do the check, go around to each server and run the command mkdir /rpmupdates.

While you are at each server, you may as well test the connection (after you make the new directory). As root, issue the following command:
mount -t nfs 192.168.1:/rpmupdates /rpmupdates

If the connection is successful, you will get your bash prompt returned with no errors or warnings.

With all the clients tested, you can now enter the following line in each /etc/fstab file:
192.168.1:/rpmupdates      /willow     nfs    rsize=8192, wsize=8192, user

Make sure the above is all one line or you’ll break fstab.

You're almost finished. Since you already ran the test on each machine, the machines should still be connected to the centralized server. Move over to the server and either copy or move all the necessary rpm files in the /rpmupdates directory. Now the clients are ready to update.

The updates
Before you make this process automated, it would be best to run a sample test. If you have rpm files ready to update (located in the server), you can go around to each client machine and issue the command:
rpm -Fvh /rpmupdates/*.rpm

If the update is successful, you'll know it by the familiar printing of the hash marks as rpm progresses through the update. A successful test means you are ready to move on to the automation process.

Automating this process is merely a matter of installing a root cron job to run (at whatever interval you like) the proper upgrade command. It is, of course, best to choose a time that will see little-to-no bandwidth usage. As root, open up your favorite editor and enter the following on one line:
30 0 * * * rpm -Fvh /rpmupdates/*.rpm| mail -s 'Output of rpmupdates' admin@yourdomain.com

(where admin@yourdomain.com is the e-mail address where you want e-mail notifications to be sent). The above line will tell cron to run the update command at 12:30 A.M. every day.

Save that file as rpm_cron and then enter the command (in the same directory you just saved rpm_cron):
crontab rpm_cron

which will install the contents of rpm_cron into root's crontab.

Once you have the crontab entry installed into each machine, you can add a second cron job to delete the old rpm files on the server (after the update is complete). To do this, move back over to the server, open up your favorite text editor again and enter the following line:
* 2 * * * rm /rpmupdates/*.rpm

After you save this as rpm_del, enter the following command (again as root and from the same directory you created the rpm_del file):
crontab rpm_del

Now every night at 2:00 A.M., the rpm files located in /rpmupdates on the server will be deleted. The reason for doing this is basically to keep the system clean and easier to administer.

You could also take this system one step further. Let's say you get your rpm files from the same location—ftp://ftp.lucky.server/rpmupdates/—every time, and that same location updates those rpm files on a nightly basis (this is asking a wee bit too much, now isn't it?). If this is the case, you could add the following line to root's crontab (on the server):
* 23 * * *  wget -r ftp://ftp.lucky.server/rpmupdates/

With the above crontab entry, the server will use the wget command to pull down all the files from the ftp.lucky.server location.

Conclusion
With some very creative scripting, you could make this system do a heck of a lot more than I’ve outlined here. A clever script kiddie might use a script to run this entire process all at once (instead of using three or so cron jobs). In fact, the tricks you can pull are fairly limitless with these types of systems.

Of course, there are proprietary applications that can do these chores with a simple click of the button. Those proprietary systems, however, are not nearly as flexible—nor do they give you the geek seal of approval as this hacked-together system does. It's your call: Win your Linux merit badge of honor and control every element of your network-wide upgrades or hide in shame as you purchase a third-party system that gets the job done but has the footprint of Microsoft Office all over it.

About Jack Wallen

Jack Wallen is an award-winning writer for TechRepublic and Linux.com. He’s an avid promoter of open source and the voice of The Android Expert. For more news about Jack Wallen, visit his website jackwallen.com.

Editor's Picks

Free Newsletters, In your Inbox