By John Locke
Linux inherited many things from UNIX, but one of the most important is the idea that while small programs perform modest functions individually, together they combine to make one powerful system. You can build up a system that meets your needs exactly by connecting and combining a multitude of individual programs, each doing its part to contribute to the whole. Setting up a killer e-mail server is a perfect example of this design philosophy.
In this article, I will begin the process of implementing a system that uses seven software packages to transport, deliver, banish, organize, and administer e-mail. You can customize this setup by substituting the software you prefer for different functions, omit or insert functions, and build up a system with exactly the features you need. This article will provide an overview of the complete system with explanations as to why the features were chosen. The system I'll implement uses the following applications:
Articles in this series
Future articles in this series will detail how to implement the system within your existing network. In upcoming articles, this e-mail system will come together with topics covering:
- SMTP and IMAP with MySQL
- Procmail, Bogofilter, and Amavis
- SquirrelMail and Web configuration
- Mailing lists
- Remote e-mail access
The heart of e-mail is Simple Mail Transfer protocol (SMTP). E-mail travels from server to server across the Internet through message transfer agents (MTAs) speaking SMTP to each other. The granddaddy of all MTAs is a program called Sendmail, which has grown organically over the years to become a powerful, confusing, easy-to-mismanage beast. Sendmail is not for the faint of heart.
Fortunately for us mortals, several alternative MTAs are now available. Three of the most popular newer MTAs are Postfix, Exim, and Qmail. I prefer Postfix because of its reputation for stability, security, ease of administration, and high performance. Postfix can deliver directly to a standard UNIX mailbox (one file with new e-mails appended) or to a Maildir folder (one directory for each mail "folder," with a single file for each e-mail).
Valid e-mail eventually ends up at the receiving MTA. That MTA must know what to do with the e-mail. The MTA hands the e-mail off to a local delivery agent (LDA), which is responsible for storing the e-mail until the user retrieves it. Postfix has its own basic LDA built in, but by configuring it to hand e-mail off to Procmail, you can continue processing it through spam filters and user rules.
Procmail is much more than a mail delivery agent. It distributes mail based on "recipes." For this system, Procmail routes incoming messages through SpamAssassin and then through user-defined recipes. Maildrop is a similar, newer LDA with what may be a more familiar-looking configuration—but Procmail is ubiquitous and works well.
The spam arms race grows ever more sophisticated. The latest technology available is Bayesian spam filtering. Rev. Thomas Bayes, an amateur mathematician in England in the mid-1700s, developed a formula for calculating probabilities based on statistics about contributing factors. Paul Graham, in a now-famous essay "A Plan for Spam," outlined a method using Bayesian statistics for collecting data about the frequency of words in a body of e-mail to create an effective content filter.
Bayesian spam filters promise more accurate spam detection, with fewer false positives (e-mail that gets classified as spam that isn't). The downside of these "intelligent" filters is that they must be "trained" with e-mail guaranteed to be either spam or not spam before they become effective.
SpamAssassin is becoming a de-facto standard for fighting spam based on specific phrases, headers, and other recognizable patterns in the e-mail. In version 2.5, SpamAssassin added Bayesian filtering to its arsenal. Since it comes preconfigured with hundreds of existing rules, it can train the Bayesian filter on known spam, and block much of the spam out there with little attention by the user.
So now the e-mail is sitting in a file somewhere on the receiving server. To read it, your user needs to connect to the server with another three-letter acronym: a mail user agent (MUA). Now I am reaching territory that just about everybody's familiar with: Outlook, Eudora, Netscape mail, and Pegasus are all MUAs. These programs connect to the server using either Internet Message Access Protocol Version 4 (IMAP4) or Post Office Protocol Version 3 (POP3).
IMAP provides a number of advantages over POP for handling e-mail. With IMAP, all mail remains on the server until the user deletes it. Users can organize their mail into different folders and access these folders from any IMAP-capable mail client. POP, on the other hand, is designed to download all mail to a single client. If your users do not always use the same computer to access their e-mail, IMAP is a great solution.
Courier-IMAP provides both POP and IMAP access to mail stored in Maildir format. It can use MySQL to define user accounts and support encrypted connections. I've found it to be much faster than the more standard UW-IMAP server—and perhaps more importantly, it recognizes "virtual users," or users who do not have a UNIX account on the server.
What's a generic database doing in this list? In this system, it's the glue that connects user accounts between Postfix and Courier-IMAP. While there are several other ways of integrating these servers, MySQL has the added benefit of being completely accessible to Web applications built with PHP. You should add a plug-in to SquirrelMail so that users can change their passwords, and build a simple Web-based user administration application.
Hotmail, Yahoo, and other Web-based e-mail services have the compelling benefit of allowing you to access your e-mail from any computer, anywhere in the world. SquirrelMail provides the same benefits to your users. It's a Web mail system that's packed with features. Besides being able to send and receive messages, your users can search for text within messages, sort messages into mail folders, access calendars and address books, and see what the current weather conditions are.
SquirrelMail uses IMAP to interact with the server, which has another enormous benefit: the mail and folders you see in SquirrelMail are the same as those you see in any other client configured to use IMAP. Therefore, you can read new e-mail in SquirrelMail while at a client, move it into an appropriate folder, and when you fire up Eudora at home, the mail is already organized. In this killer e-mail setup, I'll add some system administration capabilities into SquirrelMail so you'll be able to do more using the Web mail than you would by using ordinary mail clients.
If you have remote users, you need some way to authenticate them before allowing them to relay mail through your server. Otherwise, spammers will find your mail server and use it to create more mayhem. Pop-before-SMTP is a simple Perl daemon that monitors the system mail log and adds IP addresses to an authenticated user list, allowing users to send e-mail.
Next up: A working mail server
Now that you know the ingredients to creating your very own e-mail system, I will follow up with the details on putting it together. After working through the steps in the next article, you will have a working mail server that can send and receive e-mail. Later articles will add features to the basic mail server, until you have the killer e-mail system of your dreams.