Reconstruct corrupted mailboxes

Cyrus is an enterprise-class mail system designed for use in all sorts of standards-based environments. If you are using a Linux-based e-mail solution with IMAP mailboxes, it's quite likely that the underlying system is based on Cyrus.

Sometimes it's necessary to move mailboxes from one server to another or to recover a mailbox from backup media. As one might expect, this is rarely as simple as copying the files back into the correct directory. Due to the fact that mail is constantly flowing, it's often the case that new e-mail arrives as the archived mail is being copied back in to the mailbox. In order for Cyrus to recognise both the old and new messages in a mailbox, it needs to be reconstructed. It could also be the case that a mailbox has somehow become corrupted; this can also require reconstruction.

IMAP mailboxes

Before reconstructing a sample mailbox, let's have a quick look at the directory structure of an IMAP mailbox.

On my server, all of the IMAP mailboxes are held in /var/spool/imap/user/ with each user having his own subdirectory. If there is a period in the username (for example, john.doe) then it's replaced with a ^ in the mailbox directory name (making the directory name john^doe).

Dropping in to a user's directory, you will find numbered files ending with a period (1221. 1443. 2681. 122. and so on); there is one file per message, and all are stored in RFC 822 format.

You will notice three files in each folder, called cyrus.cache, cyrus.index, and cyrus.header-these maintain various types of information about the mailbox and its contents, including message status (unread, read, replied to, etc.). If you are transferring or recovering a whole mailbox, it's desirable to include these files. The reconstruct tool is supposed to recover information from these files to preserve message status (un/read, replied, etc.), but I've found it rarely does. Seeing all messages as unread is an unfortunate side effect of recovering or moving mailbox data.

Cyrus sees subfolders inside of a mailbox as mailboxes in their own right. The structure of these folders is exactly the same as the root mailbox with each message represented by a single file and the same index/header/cache files mentioned previously. I don't want to go in to too much detail about the inner workings of Cyrus, so for a deeper analysis I would suggest taking a look at this overview.

Reconstructing a sample mailbox

In my example, I'm moving test.user from one server to another. I have created a new account on the target server, redirected incoming mail to that server/account, created an archive of the user's mailbox on the old server, transferred that archive, checked its integrity (MD5), and then unpacked it in to the correct location on the target server.

So, how do we recover and reconstruct the mailbox? There are two ways: either from the cyradm console or using the reconstruct tool. I tend to use the latter.

First, we need to switch from root to the cyrus user:

# su - cyrus

Next locate the ‘reconstruct' tool, it should be in the default path of the cyrus user but if not use ‘find' or ‘locate'. Mine lives in /usr/lib/cyrus/bin/.

The usage of ‘reconstruct' is reasonably straightforward. I generally use the -r and -f flags. The -r flag tells the reconstruct tool to recursively reconstruct sub-mailboxes (mailbox subfolders) and the -f flag instructs it to examine the filesystem underneath the mailbox and add any directory containing a cyrus.header file. The -f flag is generally used when restoring a mailbox from backup or moving data from one mailbox to another.

cyrus@mail:~> reconstruct -r -f user/test.user




Before testing the account, it's recommended to fix the quotas:

cyrus@mail:~> quota -f

cyrus@mail:~> exit

Log on to the IMAP account via your preferred agent (Webmail, Outlook, Thunderbird) and you should be able to see all of the messages and subfolders. If you can't see any of the subfolders check that your client is subscribed to them.

Repairing and rebuilding mailboxes on a Cyrus server isn't as daunting as it may initially appear; however, it does seem to be a little quirky. I usually find that mailboxes (including subfolders) are properly imported but the message status switches back to unread. This is an unfortunate side effect and can be rather annoying. If anybody has successfully preserved this, please leave a comment and let me know.