There are three constants in life: death, taxes, and system crashes. All three may be inevitable, but at least you can recover from a system crash. While it can be a painful situation, if you have the right tools, recovery can be much easier.

Although Linux is usually known for its rock solid stability, a Linux server may still sometimes crash and/or become unbootable, often due to hardware issues. On these occasions, one of the best tools to have at your disposal is a bootable Linux CD.

Linux distribution

For the purpose of this article, I performed the steps listed below on a minimal Red Hat Linux 8.0 installation.

Take note at installation
At the end of the typical Red Hat Linux installation session, you’re asked whether you want to create a boot disk. If you are serious about system recovery in the event of a failure, don’t skip this step. This disk will be your first step in recovery should disaster strike. If you chose not to do this, you can do it later using the mkbootdisk utility from the command line.

The many tools of Linux
One of the great things about Linux and open source software in general is that there is a plethora of tools out there just waiting to be discovered. The chances are good that you will find a utility that almost perfectly suits your needs if you look hard enough. The same philosophy holds true for tools that aid in the recovery of a crashed system. Putting together a CD with all of the appropriate files and utilities would take a long time if you had to figure out exactly what to put on it. Fortunately, some folks in Belgium have made this entire process simple and straightforward by putting together a utility called mkCDrec.

What is mkCDrec?
MkCDrec (Make CD-ROM Recovery) is a utility that creates a bootable disaster recovery CD for Linux. It supports a number of file systems, RAID, and optional utilities that can be useful in the event of a failure. Among these utilities are partition management tools, RAM test tools, and data salvage tools. In addition, mkCDrec supports the most recently released versions of popular Linux distributions, such as Red Hat 8 and Mandrake 9

Getting started
The first step in creating your recovery CD is to download mkCDrec. At the time of this writing, the current version is 0.6.7 and is available for download here. I have saved the file mkCDrec_v0.6.7.tar.gz in my home directory of /home/slowe on my Red Hat Linux 8.0 server. In addition, you can download the optional mkCDrec utility package, which contains a number of useful utilities that can be burned onto the recovery CD. The utilities package is currently also at version 0.6.7. For the remainder of the installation, you will need to have root user privileges. Here are the steps:

  1. cd /home/slowe [Switch to the directory where you stored the download.]
  2. tar zxvf mkCDrec_v0.6.7.tar.gz [Expand the distribution.]
  3. cd mkcdrec [Switch to the directory created during the expansion.]
  4. tar zxvf /home/slowe/mkCDrec_v0.6.7_utils.tar.gz [Expand the utilities.]
  5. make test [This is a script that checks your system to make sure that all of the utilities required by mkCDrec are available; click here to see an example of the output of this command]

As you can see from the output, some problems on my server need to be resolved before I can continue. The nasm error refers to a utility that allows you to work with assembly code on Linux. I was unable to find this package on my Red Hat Linux 8 CD set, so I downloaded it here and installed it with the following command on my Linux server:
rpm –install nasm-0.98-8.i386.rpm

A working kernel is required since the recovery CD needs to be able to boot into an OS that can actually work with your Linux system. The mkCDrec program comes with a 2.4.18 kernel that it will use to create the recovery CD.

The other error indicates that a utility to burn a CD is not present. Since I do not have a burner on my Linux server, this makes sense. I will be burning this CD on my Windows machine, so I can take the advice of the installer and change the configuration to ignore the fact that I do not have a burner. I can accomplish this by editing the file with the command pico and then changing the line BURNCDR=y to BURNCDR=n.

After installing nasm and fixing the configuration file to tell it to ignore the burner, I can run make test again to see whether there are any more errors. In this example, I do not have any more errors, so I can continue the installation.

The next step in the installation is mostly menu driven; the menu is shown in Listing B.

Here is a summary of the menu options:

  • Option 1 creates a bootable CD with rescue utilities only. No data from the system is stored on the CD.
  • Option 2 creates a bootable CD with rescue utilities as well as backup information from the system. These can be dangerous to leave lying around, as these CDs will provide unrestricted access to the data stored on them. This option will create one or more ISO images, depending on the size of the files. They are stored in /tmp/backup.
  • Option 3 is the same as option 2 except that the ISO images are stored somewhere else instead of /tmp/backup.
  • Option 4 uses a tape for the backup instead of the CD. The rescue CD is still created.

For this example, I am going to choose option 1 so that I can copy the generated ISO file to my Windows machine to be burned onto a CD. If I wanted to back up my system files and configuration files to CD, I could choose option 2 or 3, but for simplicity and time, option 1 is fine. This will give me with a CD that I can boot and attempt to recover a crashed system but will not provide me with any backups of important files from my current Linux server.

Once I choose option 1, the mkCDrec script starts its work. This process takes a while, because the utility needs to build the entire contents of the rescue CD, including the boot loader, the kernel, and the rescue utilities. The final few lines of the process will tell you where everything is stored, as shown in Listing C.

Burning the CD
If you are new to Linux or have not delved deeply into the operating system to understand how it works, what is happening can be a little confusing. You just created a CD image file that contains a bootable Linux kernel, as well as a whole host of utilities that are extremely useful in the event of a system failure. But to be useful, this image needs to be burned to a CD on a system with a CD burner.

In my case, I just copied the ISO image file to a Windows machine where I have both Roxio Easy CD Creator and Ahead Nero CD burning software packages. Using either package, I can burn the ISO image to a CD.

At this point, I have a bootable CD that I need to test. It’s critical to test the CD you create to make sure that it actually works. Nothing would be worse than skipping this step only to find out that it won’t boot when you need it in an emergency.

Start by making sure that the BIOS on your Linux server is properly configured to support booting from CD. After you verify that, you’re ready to test the new recovery CD. Pop in the CD and restart the machine. (It is best to test this on a lab machine.) Once the disk loads, you will be presented with a special boot loader used by mkCDrec (Figure A) as well as some helpful instructions on how to proceed.

Figure A
mkCDrec boot loader

When the system finishes loading from the CD, you need to be aware of a couple of things. First, you have a bare-bones Linux system running from a RAM disk. Second, you have the mkCDrec boot CD already mounted as /cdrom.

You also have the ability to mount any devices on your Linux server, assuming mkCDrec supports them. If you don’t know the device names and file systems of your Linux server, you can get this information by using the cfdisk (configure disk) command. This will display a list of devices on your system, as shown in Figure B.

Figure B
The cfdisk utility

To mount the / (root) partition on my Linux server, I use the following command:
mount -t ext2 /dev/sda2 /mnt/local

The mount point /mnt/local now points to the root of my Linux server, and I can browse around the file system if I need to simply by changing to the /mnt/local directory. The ability to mount the file system is critical in the event that the boot loader becomes corrupt on your server and you need to fix it or offload data to a different server.

What else can be done?
Once booted with mkCDrec CD, you can do a variety of things. Most notably, if you selected the option to create a backup of your system rather than simply creating a rescue CD, you can execute /etc/recovery/, which will prompt you for the complete process in restoring your system to the state it was in when you created the CD set. Note that if you try to execute this script without having first created a backup (i.e., you created only a rescue CD), you will be told that this is the case and that there is no data to restore.

The /etc/recovery directory contains many other utilities that can help restore or fix a corrupt system. The mkCDrec Web site has details on these utilities.

If you chose to download the mkCDrec utilities and properly installed them, they’ll be available at /cdrom/utilities and include some real lifesavers, such as parted (a partition editor), memtest (a memory testing utility), and e2salvage (a utility to salvage data from a damaged ext2 partition). For more details on these additional utilities, look at the mkCDrec Web site.

Unfortunately, system crashes are a part of life. But having the right tools to recover from them can make all the difference. One of the things that I like about mkCDrec is that you can create the CD even after a crash (on a different system, of course) and have the full range of recovery utilities available in one place. And with its ability to back up your entire system onto CD(s), mkCDrec can be an invaluable a part of every Linux administrator’s disaster recovery toolkit.