Discussion on:

Rescuing a failed domain controller: Disaster recovery in action

Has anything like this ever happened to you? What else would you recommend as part of the recovery process?

The procedures outlined in the article are pretty complete. But it shoudn't have to come to this. This example stresses the importance of preparedness.
- Plan for different kinds of disasters. One DC going down in a multi-dc environment is one thing, but what do you do when you're entire DC structure gets corrupted? Think about a schema update gone bad, virusses, malicious user.
- Verify. A plan is great, but it does you no good if for whatever reason it doesn't work. So test you're backups regularly, make sure all required people know where to find backups, ADRM passwords etc.

And of course, learn from mistakes made.
my 2 cents

I joined an organisation a year ago (their first IT manager) to a number of problems, highest priority being a bad stripe on their Raid5 array.

Shortly after I started, I installed more comprehensive backup software (veritas 9) with the DR module, ensuring I had a good and verified backup each day. Their server then started failing one drive at a time. I've done disaster recovery, but never under this pressure.

In the few hours as two drives went down, I set up a second Global catalogue DC, replicated AD and joined it to the domain as a failsafe if the backup didn't hold.

Thankfully, and after a long night breaking and rebuilding the array, restoring data, it worked. Exchange mounted, and all I had to sort were a few event log problems.

Since then, I've become a serious fan of Veritas and their DR module. I get the impression I was a little lucky though, after reading some other horror stories...!

I?m an outside consultant and three of my customer have had the same bad strip in the array. All three have been with IBM servers and the fix is to rebuild the server. One of the customers had a second drive go bad. The other two were due to an array rebuild from a failed hard drive. Al l the arrays were RAID-5.

IBM 235 server running 4 drives in a RAID5 array, I was informed through our support company that this was due to a mismatch in hardware/software drivers for RAID? As you can tell, it's not my speciality..

Backups can save your life if you use it right. Use something else than the backup software that comes with Windows Server 2003. I like Backup Exec 10 or above. For the AD I will advise to do a separate backup on a CD once a week and keep it safe. DNS is also something else that needs to be considered to backup seperatly. To backup the System Volume inforamtion is good for AD and DNS, but I have found, to keep it safe on a CD, like the intro said, what if the "big" backup fails? Then what? If you have more than one forrest with multiple DC's, then a have to is roles. It gets more complicated with multiple forest and subs. But from experience, do the AD, DNS and Roles on a CD once a week, and test your backup restores! The 11th comandment - Do BACKUPS!

I hadn't considered a weekly CD backup.

Shortly looking at implementing a test strategy, anyone know any particuarly good do's and dont's?

Louis did the correct thing with the role seizure and metadata cleanup. However as being previously employed as a AD Support Engineer in Microsoft's Directory Services EPS Support group, one thing I'd like to point out to the 'error' that Louis experienced while seizing the FSMO roles: ntdsutil, by default, will ALWAYS attempt a transfer first, even when you explicitly type in the 'seize' command at the ntdsutil prompt. If he had read all the output from the command he would of noticed the output, 11 lines down saying: "Transfer of Schema FSMO failed, proceeding with seizure..." And obviously, just like a forced demotion of a DC, the AD database will not be globally updated of the changes done (such as the orphaned FSMO records) But nonetheless, he did the right thing.

Good job Louis

Good ideas, but what is applicable for a Windows 2000 Server instead of a 2003 Server?

The seizure of FSMOs didn't fail. Since the seizure operation is irreversible, NTDSUTIL will attempt to transfer the roles first. That is what you're seeing fail. It then proceeds to seize the role.