It’s not an easy task to move an enormous company. So it came as no surprise that the relocation of our company to a bigger, newer building didn’t happen as expected. For one thing, space kept being filled, and before long we were completely out of room. As a temporary measure, our accounts department was moved from the main office to a building in Greenwich and, therefore, needed computers.
Luke Mason is an IT manager for one of the largest CD, cassette, and DVD manufacturers in Britain. In this article, he describes how he recovered from a mistake while installing a Deerfield Mdaemon e-mail server.
The new office space would be populated by accountants, including the finance director. Unfortunately, the unofficial budget required us to spend as little as possible. Nobody was going to pay for new workstations, so it was a case of porting the current network setup over to a new server and using the software we already had with the odd license upgrade here and there.
Surprisingly, this part of the move went smoothly. We spent one Saturday transferring files from the old server to the new one. Only one task remained—to install the e-mail server.
It seemed like a shortcut . . .
We use Deerfield Mdaemon for our e-mail server, and then a simple MS Mail post office picks up mail from there via POP3. Mdaemon allows a set of users to be created and will then download messages from a remote POP mailbox for each user, hosted at our ISP. As the name suggests, Mdaemon hails from a UNIX past and doesn’t come with a manual. This meant that for a Windows kid like myself, with only a vague knowledge of Internet e-mail and its protocols, installation was a hassle. The simple option was to simply stick the current installation onto a Zip disk at the main office and cart it over to the new installation to be copied onto that server. Lazy? Yes. Easy? Apparently so.
The beauty of Mdaemon is that all settings are stored in data files within the installation directory; the registry is not modified in any way. Once the entire directory was copied, all that was left was to remove the users who work at the original office so mail could be collected for the right users at the right office. When this was done and tested, I walked away thinking everything was fine.
Called on the carpet
About a week later, I got a call from the financial director, who was spitting nails because Sage had stopped working. While I knew that Sage had never been a reliable piece of software, it was critical to the company. I jumped in a cab and headed over.
The error message we were getting was that Sage had run out of disk space to store its data files. As a side effect, the files were being corrupted. This had to be rubbish. A 25GB hard disk does not get filled in a week by accounting software.
I spent two hours trying to repair the files with Sage’s repair utility (which, incidentally, is extremely good). I called the Sage help desk numerous times and even attempted to restore the data files from a backup disk, but it mysteriously failed. I was left thinking that I hadn’t set ArcServeIT up properly. I decided that as a last resort, I would have a look at the hard disks.
Firing up Explorer, I checked out the free space on the data partition to find only a few kilobytes free. Looking further, I narrowed the space down to Mdaemon’s directory and then into a small subdirectory called BADMSGS. This folder is where Mdaemon puts temporary message files that can't be sent, while it is waiting to retry sending them. This folder currently contained a grand total of 35,000 files, taking up 19GB of hard disk space.
After nearly falling off my chair in surprise, then browsing the files with Notepad, I soon worked out what had happened. When I removed the user profiles from the original office, there was still a message in the system to one user whose profile had been deleted.
As the user was local and now no longer existed, Mdaemon appended a line of text onto the message, warning that it couldn’t be sent and so would be retried. It plonked it into our friend, the BADMSGS directory. This had been happening every few minutes for a week, hence the 35,000 files.
The more mathematically minded will be able to tell you that a message containing only text weighs in at just a few kilobytes. Even multiplied by 35,000, this doesn’t eat as much hard disk space as had obviously been eaten. The bigger problem was that each time this extra line of warning text had been appended, the file size had been bumped up by one kilobyte or so, taking the original message from under 2 KB to over 3500 KB! This exponential scale is what had munched on the storage for a week. As Sage had no more space to write its files, these had been “corrupted”—or only half written. The unraveled mystery also explained why ArcServeIT hadn’t restored the backup files. There was no disk space to put them in.
After a quick touch of the [Delete] key, and 30 seconds or so, Windows deleted 35,000 files. All that was left was to prevent Mdaemon from placing any more messages in the BADMSGS directory and to repair the corrupted Sage data files. Luckily, they weren’t missing much, as the accounts users had called me when they first noticed the problem.
A mistake leads to humility
I have to take a portion of the blame myself for setting up server software without knowing how it worked. I should have trusted Sage and checked the disk space first of all, but how often do you get ridiculous error messages that couldn’t possibly be true? (Think of 16-bit DOS applications under NT 4 telling you they don’t have enough memory to run on a 128MB workstation.)
And why doesn't Mdaemon have some kind of checking to make sure it's not using too much storage? Why did it try to resend that message so many times? What had happened was akin to the most rudimentary virus payload—achieving in a week what no virus has managed to do to our system in the last few years.
Why on earth didn’t ArcServeIT tell me that there was no disk space to restore the files? A backup system should surely take this into account rather than simply placing an entry in the log file stating, “Restore operation failed.” There was no onscreen error message at all during the entire process, and nothing alluded to the lack of disk space whatsoever.
Hindsight is 20/20
What would I do differently next time?
- Make sure that I knew enough about Mdaemon to carry out a clean install.
- Test and configure the system thoroughly before even taking it to the other office.
- Not be so afraid to disagree with tight deadlines when I need more time for a project.
Although I ranted about Mdaemon, I would recommend it for similar purposes. It’s cheap, and it has all of the features required to implement an Internet mail system that doesn’t need too many bells and whistles. Luckily, the time I lost was minimal—and the lessons I learned were valuable.
Experience is the best teacher. Tell us about a project that taught you an unexpected lesson. Post a comment below or send us some mail.