Your Exchange server has crashed hard, taking Windows down with it. What do you do? Weep softly? Work on your resume'? Nope. Check out this article by Brien Posey instead.
Imagine for a moment that it's late at night and you are blissfully asleep. Suddenly, you are woken by the sound of your pager. You look at the page and notice that it is in relation to a problem with one of your Exchange Servers. You groggily get out of bed and connect through the company's VPN to the network to check the server's status, but the server isn't even showing up. You begrudgingly throw on some clothes and drive to the office, expecting to have to reboot the server. However, upon your arrival you discover that the server's system partition has been completely corrupted. You can't even boot Windows, much less run Exchange. What do you do?
Ideally, you would simply restore the full system backup that you made a couple of hours before the crash occurred, but in the real world things are rarely that easy. Over the last several years, I have performed dozens of Exchange restorations, and no two of them have been exactly alike. The key to being able to restore Exchange to a functional state is to have a thorough understanding of the various Exchange database files and how they relate to each other. You must also understand Exchange Server's dependency on Active Directory.
Before I get started, I just want to say that this article assumes that you are trying to restore an Exchange 2003 Server running on a Windows 2003 operating system. Some of the techniques that I will be discussing may not work if Exchange is running on Windows 2000 Server.
OK, I want for you to completely forget about Exchange Server for a while. I know that you need to get Exchange back up, but remember that Exchange has some dependencies. Before you can run Exchange Server, Windows must be functional as must Active Directory and IIS. Therefore, step number one is to return Windows to a functional state. Unfortunately, this means more than just reinstalling Windows though. You can't just reinstall Windows, restore Exchange, and expect it to work. You have to return Windows to the state that Exchange Server is expecting.
Hopefully, you have a full, system state backup of the server. If you don't have a system state backup that's OK though. You can still recover Exchange without a system state backup as long as Exchange isn't running on your organization's only domain controller, or something crazy like that. If Exchange was running on a domain controller, but there are other domain controllers in the domain, you must check to make sure that one of the remaining domain controllers is a Global Catalog server and that the network has a functional DNS server.
For the purposes of this article, I am going to assume that you have a system state backup available to you. That being the case, there are two different methods that you can use to recover the server. The traditional method involves installing Windows and then restoring the backup. The other method involves using something that's new to Windows Server 2003 called Automated System Restore. In the sections below, I will talk you through both methods.
Automated System Recovery
As you probably know, manually installing Windows and then restoring a backup tape can take forever. If you are trying to restore a failed server, you probably don't want to wait that long. If it's the middle of the work day, you want to get the server back up fast so that everyone will quit screaming at you. If it's the middle of the night, you want to restore the server quickly so that you can go home. This is where Automated System Recovery (ASR) comes in.
The idea behind ASR is that you don't have to manually
install Windows. You can tell the Windows Setup program to rebuild Windows off
To make an ASR floppy disk, take the backup tape that you are going to be restoring to another server. Now, you will want to use NTBACKUP to restore two specific files. However, rather than restoring these files to the hard drive, you will restore them to a floppy disk. That disk will then become your ASR floppy. The files that you must restore are both located in the \Windows\repair folder. They are named asr.sif and asrpnp.sif.
Now, insert the backup tape into the failed server and boot the server off of the Windows installation CD. When prompted, select the ASR option and insert your ASR floppy. Setup will now restore your backup. One word of caution though is that running an ASR based recovery causes Windows to delete and recreate the various partitions on your server hard disks.
Restoring using Windows Backup
Restoring the server by using NTBACKUP tends to be more work than using ASR. The first step in the process is to create a partition that you can set Windows up in. The partition should be about the same size as what was previously defined. Later, after Windows is up and running, you will have to create any other partitions that existed previously. Once you have created the necessary partition, make sure and format it with the same file system that was being used previously.
Now, it's time to reload Windows. A lot of people don't realize it, but there is a certain etiquette to reloading Windows when you are going to be restoring a backup on top of it. One of the first things that you will have to address during the reinstallation process is the computer name. If you are going to be restoring a system state backup, you can call the system anything that you want, because the restore operation will overwrite the computer name. What ever you decide to call the server, just make sure that you don't attempt to join a domain.
If you aren't going to be performing a system state restore then you will have to give the computer exactly the same name as it had previously. Before you do though, you will have to use Active Directory Users And Computers console to remove the previous instance of the server from the domain (assuming that domain controllers still exist). Otherwise, you won't be able to join the domain later on because the server's name already exists within Active Directory.
The next issue that you will have to address is which components you want to install. There's a big misconception that if you are going to be performing a system state restore then you don't have to put any thought into your initial Windows installation. This simply isn't true though, especially if you are going to be installing Exchange Server later on.
The biggest thing to remember when you are choosing which components to install is to make sure that you install IIS, and the NNTP option. It has been my experience that if you don't specifically install these components up front, that Exchange won't recognize them properly unless you install them and then perform the restoration a second time. It's a whole lot easier to just install these components as you install Windows than to have to deal with that mess later on.
Once Windows is up and running, use the Disk Management console (DISKMGMT.MSC) to create any remaining partitions that don't presently exist. These partitions must be formatted with the same file system as was used prior to the crash. Next, you must install any service packs and hot fixes that existed on the server at the time that the backup was made.
Did you catch that? You have to revert the system to the service pack and hot fix level that was in use at the time that the backup was made; not what was in use at the time of the crash. This is extremely important. Sometimes you can get away with missing a hot fix or applying an extra hot fix, but you have to have the right service pack. Otherwise the backup won't restore correctly and you will have to reformat the system and start over again.
Now it's time to restore the backup. This is another step where it is easy to make a mistake. A lot of people assume that you should just perform a full system restore. If the server that you were repairing was just a file server then this would be the correct course of action. However, you must keep in mind that you are restoring an Exchange Server. The backup tape is probably chalked full of Exchange related data. Restoring anything Exchange related at this point in the game would be a big mistake in most cases. Therefore, you should go through the listing of files to be restored and deselect anything related to Exchange Server. This includes any folders that contain databases, transaction logs, and most of all, the \Program Files\Exchsrve folder.
Once you have deselected the various Exchange Server components, let the restore operation run. When the restore process finishes, reboot the server. When Windows boots, you will see a message indicating that one or more services have failed to start. The reason why you are getting this message is because the system registry is full of references to Exchange Server, but you have not yet restored Exchange Server.
Right now you might be wondering why I didn't have you to restore Exchange Server. There are a couple of reasons. One reason is that for most people, the backup tape that contains the system state backup of Windows is not the same tape that contains the most recent Exchange Server backup. If you were to restore Exchange from an older tape, even though a newer tape is available, you could confuse other Exchange Servers in your organization.
Let's say for the sake of argument though that you made the tape the night before the crash and that the tape contains a system state backup and the latest Exchange backup. It is still a good idea to avoid restoring Exchange until after Windows is up and running. The reason for that is because in order to perform an online restore of an Exchange database, your backup software must be Exchange aware. NTBACKUP is not Exchange aware by default. It only becomes Exchange aware after you install Exchange.
If you had attempted to do a full system restore, NTBACKUP wouldn't know that it was supposed to do an online restore. Therefore, Exchange and its databases would have been restored at the file level. This almost always results in the Exchange databases being in an inconsistent state. If that happened, you would have to use some of Exchange Server's built in utilities to bring the databases back to a consistent state so that they could be mounted.
Since Windows is now up and running, but you haven't done anything with Exchange yet, I recommend taking a few moments to make sure that Windows is functioning properly before you restore Exchange Server. First, check to make sure that the machine has received the correct computer name. You might also log out and verify that you were logged into the domain, and not into the individual machine, as a way of testing domain connectivity. Next, open a Command Prompt window and enter the IPCONFIG /ALL command. This will allow you to verify that the machine has the correct IP address and that TCP/IP is configured correctly.
If it appears that TCP/IP is configured correctly, then enter the NETDIAG command followed by the DCDIAG command. These commands will test the machine's network connectivity and its ability to communicate with domain controllers respectively. If either of these commands return errors then re-enter the command, but append the /FIX switch to fix the problem.
Once you are convinced that Windows is running properly and is communicating with the rest of your network successfully, it's time to restore Exchange Server. Remember what I said though. NTBACKUP can't restore Exchange Server until it becomes Exchange aware, and it doesn't become Exchange aware until you install Exchange. There is a way around this Catch 22 though.
The way to get around this problem is to insert your Exchange Server installation CD and manually run the Setup program with the /disasterrecovery switch. What this switch does is forces Setup to install Exchange Server without any databases or database related components. That way, NTBACKUP will be updated so as to allow you to restore your Exchange databases, but you won't encounter any of the tricky issues involved in overwriting existing database components.
Before you restore your Exchange databases though, you must update Exchange Server with any service packs and hot fixes that existed on the server at the time that the backup was made (the Exchange backup, not the Windows backup). After doing so, run NTBACKUP and tell it to restore your Exchange organization. Be sure to use the Exchange option rather than restoring individual files. Otherwise you could run into the same types of consistency problems that I talked about earlier.
In this article I explained that if your server completely dies, it takes a lot of work to get it back up and running, even if you have a full backup available. I then went on to discuss several areas where people frequently make mistakes when restoring an Exchange Server to operational status.