SolutionBase: E-mail sender authentication: Is it the solution to spam?

It's getting to the point where spam is crippling e-mail, making it almost useless. Here's how sender authentication technology using Sender Policy Framework and Microsoft's Sender ID can help get rid of unwanted e-mail.

Life is such that soon after someone comes up with a good thing, someone else comes up with a way to ruin it, and e-mail is no exception. Spam or UCE (unwanted commercial e-mail) has proliferated out of control. Literally thousands of spam messages flood into the typical company's mail server; luckily, good spam filtering software can prevent most of them from finding their way to users' mailboxes. Laws have been passed with the goal of curbing the spam tide, but they are difficult to enforce, in part because spammers often disguise the real source of their messages by forging return addresses and e-mail headers.

An even more dangerous development on the e-mail front is "phishing." Unlike traditional spam that merely attempts to sell a product (albeit through annoying and sometimes offensive means), phishing messages are fraudulent, purporting to come from the recipient's bank, credit card company or a company with whom he does business and attempting to entice the recipient into revealing confidential information such as passwords, account numbers, or social security numbers. The phisher then uses that information for identity theft and other criminal purposes.

Because of the growing amount of "forged" e-mail from spammers and phishers, a mechanism for authenticating the senders of messages is growing more desirable. Toward that end, technologies have been developed to provide a way to verify senders' identities. In this article, we take a look at the Sender Policy Framework (SPF), a platform-independent sender authentication technology, and we also discuss Microsoft's Sender ID, which is based on SPF (and which has been declared dead in the water by more than one industry pundit).ï¿?? Then we look at where cryptographic solutions (digital signing) fit into the picture.

The trouble with e-mail

Despite its usefulness and popularity, e-mail poses a couple of big problems. The system is based on the Simple Mail Transport Protocol (SMTP), which was designed with speedy delivery, rather than security, as its prime objective. In that way, e-mail is modeled on the "snail mail" system. Anyone can drop a letter in a physical mailbox with no return address or false return address information. Likewise, e-mail can be sent with an inaccurate or "spoofed" return address.

A major "loophole" in SMTP is its lack of authentication; it is, after all, the Simple Mail Transport Protocol, and at the time it was created, security was much less a concern than it is today and spam and phishing were nonexistent. Thus, there's no mechanism for detecting spoofed header information.

A three-pronged solution

It's important to note that authentication will not, by itself, stop spam. The best it can do is help stop the falsification of sender information. Spammers can use authentication mechanisms, too. Just because the sender is authenticated, that does not mean the message is one that you want. Spammers can register domains and publish SPF records just like anyone else, and in fact, a number of studies have indicated that spammers are early adopters of the authentication schemes. Thus, authenticating the sender is only the first step in stopping spam and phishing messages.

The Aspen Policy Institute came up with the "Accountable Net" framework (also referred to as the Aspen framework in Internet circles) in December 2003. This outlines a community-based approach to verifying identity and fostering accountability on the Internet, based on three factors:

  • Authentication
  • Reputation
  • Accreditation


Most e-mail sender authentication systems rely on those who own domains to publish the servers or e-mail addresses from which legitimate mail from that domain can be sent. These lists of legitimate address-domain correlations are than checked when a message arrives. If the sending address matches the address that is related to that domain in the list, it is authenticated. If the mail came from a server that is not listed by the domain owner as a legitimate sender for that domain, authentication fails.

Note that a big drawback of these IP-based authentication schemes is that only the domain is validated (not the user). SPF authenticates using the return-path (envelope sender) in the MAIL FROM field, not the From field in the header. One way to verify the identity of the individual author of the message is through digital signatures (cryptographic authentication). The most effective approach is to combine IP and cryptographic technologies.


Reputation refers to the sender's known history. Once the sender is authenticated, the next step is to determine if the sender is a known spammer, a known legitimate sender, or a "mystery sender" (legitimacy unknown). Although a spammer's domain might be using SPF to authenticate, as it sends spam it will acquire a "bad rep."

Blacklists are a form of reputation system. The problem with published blacklists in the past has been the high rate of false positives. For example:

  • Addresses were deliberately reported as spammers when they aren't. What better way to disrupt the life of someone you don't like than to get his name on the spam blacklists so his mail won't get through to his friends and business associates?
  • Entire domains were blocked as spam sources when only a single user from that ISP sent spam.
  • Spammers forged the addresses of legitimate users as their return addresses, and the legitimate users' addresses ended up on the blacklists.

This last situation, at least, would be helped by widespread use of sender authentication and blacklists would have more credibility.


Accreditation takes reputation a step further, allowing a third party (accreditation provider) to vouch for the reputation of senders, based on sophisticated reputation analysis. Accreditation services may require that senders pay to be listed and may be backed by a financial bond.

Accreditation creates another element of reputation, the whitelist, which contains "known good" senders.

SPF: What's it all about?

SPF is the most popular sender authentication technology. It was developed as an extension to SMTP in 2003 by a group led by Meng Weng Wong, who founded and runs the SPF Web site. It was originally called Sender Permitted From but later morphed into the Sender Policy Framework. The concept grew out of two previous sender authentication proposals: Reverse MX (RMX) and the Designated Mailers Protocol (DMP), and is based on the original ideas of Jim Miller, Paul Vixie and Hadmut Danisch.

How SPF works

A mail transfer agent (MTA) is the mail server software that handles delivery of e-mail messages (for example, Exchange or Sendmail). Under the Sender Policy Framework, owners of e-mail domains authorize specific MTAs (mail servers) to be the designated senders for their domains. They do this by publishing the information in the DNS records in TXT format. You need to publish an SPF record for each domain and subdomain or hostname that has an A or MX record in DNS. You must use the DNS server that hosts your domain on the Internet (rather than a local DNS server) to publish the SPF records.

The MAIL FROM command initiates SMTP messages. The return-path address is part of the MAIL FROM command, which is shown in the e-mail headers. SPF compares the client IP address with the SPF record for the domain that's shown in the return-path address to make sure they match.

You can view the Internet headers in most e-mail clients, as shown in Figure A.

Figure A

SPF checks the domain in the return-path address against the client IP address

Once the identity of the sender has been authenticated, MTAs can use reputation and accreditation (DNS blacklists and whitelists, or DNSBLs and DNSWLs) to decide whether to block or allow the message.

Future plans for SPF include mail user agents (MUAs), which we normally think of as e-mail clients such as Outlook or Eudora, to be able to recognize messages that have passed the tests of authentication and reputation/accreditation and mark them or route them to a folder indicating that they are highly likely to be legitimate mail (the opposite of the "junk mail" folder where e-mail clients currently put messages that are likely to be spam).

Drawbacks of SPF

Once widely adopted, SPF would operate on a "guilty until proven innocent" philosophy in regard to e-mail. The default would be to consider messages that are not authenticated or don't pass the reputation/accreditation test to be spam. The danger in this would be the same as current overly aggressive spam content filters: the likelihood of false positives. Receiving mail you don't want is annoying and can even be costly in terms of time and lost productivity. But not receiving a critical message could cost much more.

Keep in mind that the information in DNS is available to the public. You might have internal servers that need to send mail from your domain, but you don't want those machines' addresses published in the public records. In that case, the simplest solution is put the servers' addresses in a DNSWL. This list is not available to the public, just to your SMTP server.

Unfortunately, SPF can cause problems with forwarding. To use SPF, the forwarding MTA has to rewrite the sender address. This can be fixed by using the Sender Rewriting Scheme (SRS).

Sender ID: The Microsoft solution

In early 2004, Microsoft unveiled an anti-spam initiative based on a technology they called Caller ID for E-mail and implemented a pilot program for Hotmail. Then in the summer of 2004, Microsoft merged its product with SPF to form a new authentication specification called Sender ID.

Sender ID is based on SPF but there are some differences. To use Sender ID, domain owners would publish the IP addresses of their mail servers in DNS using XML files rather than the TXT format used by SPF (however, Sender ID is backwardly compatible with already-published TXT files used by SPF). Sender ID originally used the Purported Responsible Address (PRA) instead of the MAIL FROM return-path address. The PRA is extracted from RFC 2822 headers; in other words, it's the address that users see as the sender in their e-mail client software. The return-path address and PRA address are the same on many messages, but they can be different.

A later submission of the Sender ID specifications to the IETF allows using both the return-path address and the PRA could be used. If the return-path address is checked, it's called SPF classic mode, and if the PRA is checked, it's called PRA mode.

Microsoft applied for a patent on the PRA checking portion of the technology, and this resulted in much controversy and a number of organizations that had formerly endorsed Sender ID dropping their support.

From the perspective of domain owners, if messages with different return-path and PRA messages need to be sent from their domains, they can create PRA (SPF2.0) records in addition to the SPF (SPF1). Most domain owners don't need to worry about this, because Sender ID checks the SPF1 records for both the return-path and PRA addresses by default.

Cryptographic solutions

A different approach to authenticating senders of messages is to implement digital signatures on the message content. Most of these methods are dependent on a Public Key Infrastructure (PKI) to issue public/private key pairs. The signing is done by the MTA and the senders publish their public keys in DNS.

Digital signature technologies

There are a number of different technologies that can be used to digitally sign messages:

  • PGP (Pretty Good Privacy) can be used to sign the body of the message. Keys can be self-signed.
  • S/MIME (Secure Multi-purpose Internet Mail Extension) can be used to sign the message body. Keys are signed by a certification authority (CA), a trusted third party.
  • DomainKeys (Yahoo) can be used to sign the message body and headers.

Advantages and disadvantages of cryptographic solutions

Cryptographic solutions have an advantage over IP-based sender authentication methods in that they are able to handle forwarding. The IP-based methods will incorrectly flag a forwarded message as a forged one (unless the forwarder rewrites the return-path address).

On the other hand, because most cryptographic solutions sign the message body, messages may be incorrectly identified as forgeries if the content of the message changes after the signature is applied. This creates a problem with mailing lists that may change the message content in transit.

The final solution

There has been a great deal of skepticism and criticism of SPF/Sender ID within the industry. Many pundits question the value of IP-based authentication of sender domains in stopping spam. But these technologies were never intended to be "spam busters" in and of themselves.

Wong and others involved in developing sender authentication solutions suggest that a combination of IP-based and cryptographic methods is the best way to verify the identities of e-mail senders, and they stress that authentication is only the first step in controlling spam; it must be combined with other mechanisms such as reputation and accreditation to be effective.

It's likely that spam--like the junk brochures in our snail mailboxes--will always be with us to some degree, but thwarting spammers' efforts to disguise their identities through sender authentication and development of accurate, reliable blacklists and whitelists would go a long way toward making the problem more manageable.

By Deb Shinder

Debra Littlejohn Shinder, MCSE, MVP is a technology consultant, trainer, and writer who has authored a number of books on computer operating systems, networking, and security. Deb is a tech editor, developmental editor, and contributor to over 20 add...