E-mail spam has quickly become a major headache for anyone with an e-mail address. The total cost to the business world in terms of time and effort spent fighting the spread of mass junk e-mail distribution is astronomical—easily in the tens of billions. For the IT professional on the front line of this battle, there are numerous filters and prevention methods at their disposal. However, the spammers are ingenious and continue to evolve and refine their mass distribution techniques in an effort to bypass whatever protections may be deployed.
This continuing deploy/circumvent cycle has lead to what can easily be described as an arms race between e-mail administrators, ISPs, and other IT professionals versus the spam marketers and their shady tactics. With this escalating battle as a backdrop, authors Bryan Costales and Marcia Flynt wrote sendmail Milters: A Guide to Fighting Spam, to help IT professionals get the upper hand in their prevention efforts. A downloadable PDF of Chapter 3, Setting Up a Bait Machine, from that book is available from the TechRepublic Download Center.
In the following interview, the authors discuss the spam problem and some to the prevention techniques they have found to be most effective.
[TechRepublic] For the TechRepublic community members, who may not have hands-on experience with Sendmail servers, please explain, in general, how sendmail Milters work.
[Costales and Flynt] A sendmail Milter (Mail Filter) is a stand-alone program that runs alongside sendmail. Because a Milter is a multi-threaded application, one Milter may support many forked sendmail processes running in parallel.
Milters and sendmail processes can be put together like tinker-toys. You may mix and match them in almost any manner to construct a complex mail filtering environment from simple components.
A Milter generally screens all or part of any mail message received and advises sendmail about each part. A Milter can "accept" a message or "reject" it, or "continue"—that is, neither accept nor reject the message at the present time. A rejected message can be "tempfailed" (the sender told to try again later), "bounced" (rejected outright), or "quarantined" (held in the queue pending human review).
A Milter may also add and remove recipients, add and remove headers, and replace all or part of the message body.
Multiple Milters can run on a single server. Each is called in turn for each category of each message. For example, all Milters review the connection and HELO information before any single Milter reviews any other part of a message. Milters are called in the order defined by a sendmail configuration file. When sendmail is configured to use a Milter, it sends each bit of connection information to the Milter. Among the information sent can be connection, recipient, sender, host, header and body information.
When multiple sendmail processes are used in different roles (for example, a queue processor and a listening server daemon), each can share the same pool of Milters, but each might call them in a different order or use different Milters from the pool.
Because Milters are so versatile, they are necessarily, and unfortunately, somewhat complex. In our book, we try to make them very understandable.
[TechRepublic] In your book, you discuss spam and spam purveyors as moving targets because the technology and techniques they use to bypass filters and other prevention methods is continually refined and improved. Does this mean that, by its very nature, spam proliferation can never really be completely eradicated, but merely slowed to a manageable level?
[Costales and Flynt] Spammers are in it for the money and will go to remarkable ends to achieve their goal. To illustrate, consider ISPs that white list based on a list IP numbers. A spammer, when faced with such a site, will remain well behaved for no matter how long it takes to be promoted to a favorable place in that list. The wait can take weeks or even months, but the difference between an unfavorable IP classification and a favorable one could mean tens of thousands of dollars. Thus, the spammer is content to remain inactive until a favorable classification is achieved, then the spammer will awaken and send hundreds of millions of message. Sure, that address will be shut down quickly, but not before the spammer has gained a great reward.
So long as it is possible to make money by abusing e-mail, spammers (and today, phishers) will never be discouraged. Just as laws against crime cannot eliminate crime, laws against spam can never eliminate spam.
[TechRepublic] At the beginning of your book you mention several other methods for fighting the proliferation of spam besides sendmail Milters, including Domain Name Services (DNS), Bayesian filters, screened URLs, and postage. Do you believe that the best way to fight spam is use a combination of these methods? Is there a combination that you believe works best? What new and improved methods are on the horizon?
[Costales and Flynt] No single solution can eliminate all spam. Clearly, given the wily nature of spammers, a broad based approach is always best. Try simple spam fighting methods first to pick the low hanging fruit. Some spamming sites, for example, send all spam from a fixed IP address. Block that address (using sendmail's/etc/mail/access database) and a significant portion of your daily spam can be quickly eliminated. Follow this kind of IP screening with Bayesian and URL screening and (with daily maintenance) 90% of spam can routinely be eliminated.
The trouble is, the better you get at removing spam, the more likely it becomes that you will wrongly reject valid email. False positives are the bane of spam filtering. To rectify this problem, consider saving rejected email for periodic human review.
Two current approaches to filtering spam use sender identification and site validation. The theory used by both methods is that spammers need to hide to be effective. Sender identification seeks to make the source of spam easier to find. Domain Keys and similar schemes seek to digitally sign email so that its source can be conveniently tracked. Neither will suppress spam directly, but may embarrass some spammers out of existence.
The right solution for your site depends how much spam and phishing affects your site and your site's customers. What does the problem cost and how much are you willing to pay to reduce it?
[TechRepublic] Because sendmail Milters is open source, what advantages does it have over proprietary solutions, especially proprietary server-side filters? What disadvantages?
[Costales and Flynt] Because they are open-source, Milters are more cost effective than proprietary solutions. Many commercial proprietary solutions charge by the mailbox (some over $1 per mailbox) and thus can become very expensive. Although there is an initial investment to write Milters from scratch, maintenance of Milters typically is less costly than proprietary solutions. Additionally, as Milters become more popular for fighting spam, more Milters will be available for clients to download for free.
Although Milters were created to fight spam, they can also be used to solve many problems:
- A Milter makes it possible to archive a copy of all sent and received email (perhaps for security conformance).
- A Milter can split out attachments and store them centrally, thereby reducing duplication and lowering storage costs.
- A Milter can defer acceptance of non-company email until after business hours.
- A Milter can gather and report email statistics for use in relational databases.
Over the next few months, we expect many more Milters to become available from sites such as www.milter.org and spambook.bcx.com. As spammers become more sophisticated, the open source community will insure solutions will become readily available. Because this community is very large, getting Milter support can also be both timely and cost effective. These are just a few of the many reasons we feel open source Milters are more advantageous than proprietary solutions for all clients regardless of their size.
For a more in-depth look at what you can do to help your organization prevent the spread of spam, download the book excerpt from sendmail Milters: A Guide for Fighting Spam–Chapter 3, Setting Up a Bait Machine.
Mark W. Kaelin has been writing and editing stories about the IT industry, gadgets, finance, accounting, and tech-life for more than 25 years. Most recently, he has been a regular contributor to BreakingModern.com, aNewDomain.net, and TechRepublic.