E-mail is more pervasive these days than the business letter was just a couple of decades ago. Having a reliable e-mail server is crucial to most businesses. However, many companies—particularly smaller ones—have not taken the next step to host their own mail servers. Some rely on their local ISP to provide e-mail services, while others turn to Hotmail, Yahoo, or other global mail service providers. Hosting your own mail server offers some nice advantages, whether you need to support hundreds of users or just want to put in your own mail server in a small office. In this Daily Drill Down, I’ll take you through the ins and outs of setting up your own mail server.

An e-mail server to call your own
Most national and local Internet service providers don’t offer a lot of features when it comes to their mail systems. The majority of them give you the ability to filter your messages so you can try to block all those annoying messages for herbal diet plans, debt consolidation, and hot stocks, not to mention all those unsolicited messages you wouldn’t want your dear old granny to see. But that’s about where the level of service ends. If you want features such as out-of-office replies, automatic responses, unlimited mailboxes, and mail forwarding, you need to take matters into your own hands and put in your own mail server.

You might think that putting in your own mail server means a big outlay in new equipment, expensive software, and time to manage it all. As long as you have an always-on, broadband Internet connection, however, it can be a snap. Because many of the mail server applications run on Windows 9x and other nonserver operating systems, you may not even need to change your existing systems. Even if you don’t have a dedicated Internet connection, you can still install a mail server that dials your ISP to send and receive messages.

What you do need, however, is an understanding of how e-mail messages get routed on the Internet, how to make those messages come and go through your server, what features to look for in a mail server, and then how to put it all together into a working package. Before I get into the nuts and bolts topics, let’s take a look at what you can expect from your own mail server.

What will you gain or lose?
One of the main things you’ll gain from setting up your own mail server is complete control over how many mailboxes you can have, how those mailboxes are set up, and how they work. Are you running into size limitations on your existing mail service? Does your current mailbox keep filling up or rejecting your messages because they are too large? No problem! When you own the mail server, you can do whatever you want. Go right ahead and send that 20 MB attachment to the remote office.

That brings up another important advantage that your own mail server can provide. You can extend the benefits to others, hosting accounts for other branches or even partners. That’s particularly nice if you’ve succeeded in securing a domain name for your company. The downside comes if your remote sites and business partners start to rely a little too much on their mail accounts. If your Internet connection goes down for a few days for reasons beyond your control, or your server suddenly develops a bad case of virus-of-the-week, you don’t want people calling at 2 A.M. to complain about it. I’ve been in the 24/7 support business and hated it. You will, too, if you get in that position. If you open your server to others, make sure they understand that they get what they pay for and should have a backup option, such as Hotmail or Yahoo. You should have a backup for your own e-mail account as well.

Autoresponders are another handy feature offered by many mail servers, and they’re much like out-of-office replies. For example, maybe you have a document you want people to be able to obtain simply by sending a message to a specific account. A customer can send a message to fudge@yourdomain.whatever and get back a reply with your favorite fudge recipe. Businesses often use autoresponders to distribute information about products and services. Whether you manage services for a commercial venture or run a small home business, autoresponders might add a new way for you to interact with your customers.

Many mail servers offer features that make it easier to distribute mail to groups of people. While you can create distribution lists with any e-mail client, creating groups at the server lets anyone send mail to that group through a single e-mail address. Some mail servers give you the ability to send messages to group members in round-robin fashion. This means that each new message gets sent to a different person in the group. This is a great way to distribute messages evenly across the group, and it is typically used to distribute sales or support requests.

Keeping out spam is another potential benefit to hosting your own mail server. While you can create rules in your e-mail client to delete messages from specific senders, keeping out spam for several mail accounts can be a headache, particularly if your current service doesn’t give you any spam-blocking features. With antispamming built into the server, you can block mail from domains or specific senders for all accounts.

Many mail servers also provide built-in virus scanning or can use add-ins to scan messages coming and going through the server. You might use this in conjunction with client virus-scanning software to add another layer of security for your home network.

Laying the groundwork
To set up your own e-mail server and keep it working, you need some understanding of how e-mail gets routed across the Internet. Simple Mail Transport Protocol (SMTP) is the glue that binds Internet messaging. Mail servers use this protocol to communicate with one another, and e-mail clients use it to send messages. For example, let’s say I send a message to Uncle Ned asking for a copy of his pickled onion recipe. My mail program (let’s assume Outlook Express) connects to the mail server for my mail account using SMTP commands.

Through these commands, my message gets placed in the server’s outgoing mail queue. The server then looks at the address for the message to determine the destination. It queries the DNS to find the IP address of the mail server that serves that destination domain. My server then uses SMTP to communicate with the other server to deliver the message. If a network or server problem prevents the two servers from transferring the message, the message remains in the sending server’s outgoing queue for a specific period of time (the default is usually four days). The server periodically tries to resend the message, and if it can’t do so by the time the specified expiration period occurs, it returns the message to the sender with a nondelivery receipt (NDR).

When it comes time for Uncle Ned to retrieve the message, he has a handful of options. The most common is the POP3 protocol. When you connect to a mailbox on an ISP’s mail server, the mail program uses POP3 to retrieve the message. With POP3, messages are downloaded to your local computer, but you can optionally leave a copy of the messages on the server. This lets you also retrieve the messages from another computer, if needed.

IMAP is another protocol supported by many mail servers. With IMAP, the messages remain on the server and you access them live. You can read and delete messages from the server as well as create new ones. However, new messages are sent using SMTP rather than IMAP. IMAP is a good option when you need to access mail from more than one computer, because the mail is always available on the server instead of being downloaded to your computer—you don’t have to worry about synchronizing mail stores on different computers. However, it does require a mail client, such as Outlook Express, Outlook, or Eudora, that supports IMAP accounts.

Some mail servers also support HTTP, the same protocol used to serve up Web pages. HTTP support lets you send and receive e-mail through your Web browser. Yahoo and Hotmail are two examples of services that provide HTTP-based e-mail, and Outlook Web Access (OWA) under Exchange Server also offers this capability. The main advantage to using HTTP is that you don’t need a dedicated e-mail client program but can instead rely on your Web browser to send and receive e-mail. This makes it easy to work with your messages from any computer.

Choosing the right software
I owned and operated an ISP for several years and used Microsoft Exchange Server to provide e-mail to our customers. As you may have guessed, I instinctively recommend Exchange Server for new e-mail servers. However, Exchange Server lists for just under $700 without any client licenses and just under $1,300 with five licenses, so that puts it out of range of most small businesses.

If Exchange Server seems like too much horsepower or too much money, there are lots of other good commercial, shareware, and even freeware mail server programs that range in price from free to a few hundred dollars. A search of popular download sites such as CNET Shareware.com, Tucows, and others will turn up a couple dozen e-mail server programs.

When you’re looking for a mail server to use in your business or home office, start by weeding out the ones that won’t run on your operating system. In most cases, programs designed to run on Windows NT or Windows 2000 will also run on Windows XP, but check with the vendor to be sure. Also, most of the mail servers for Windows NT run on both Workstation and Server, while most of the ones written for Windows 2000 run on both Professional and Server.

Next, decide which protocols and features you want from your mail server. POP3 and SMTP support are a given, but if you want HTTP support, your choices will be more limited since that’s one feature that many mail servers don’t have. Still, there are servers available that support HTTP, and they don’t cost an arm and a leg. The convenience of being able to send and receive messages through a browser might offset the additional cost for a server that supports HTTP or an HTTP add-on.

Once you’ve narrowed your selection based on protocols, you can start looking at the other features you need. Some mail servers function mainly as mail gateways to your existing ISP mail server and don’t act as stand-alone mail servers for direct e-mail delivery from the Internet. Others offer both capabilities, functioning as a stand-alone mail server while also retrieving your mail from other mail services.

Next, look at special account features such as support for autoresponders, mail groups, the ability to function as a mailing list server, spam filtering, virus scanning, and any other advanced features you’d like to have.

Putting it all together
One aspect I haven’t discussed yet is how you hook up your server to the Internet. You should become familiar with the process before you buy software and get too far into making any system changes.

First, you’ll need your own domain if you want to receive messages directly to your server. You can register your domain at Network Solutions, Register.com, Go Daddy, WebSite.ws and through other domain providers. When you register a domain, you need to provide the IP address and host name of at least one DNS server for your domain, although most registration services require two DNS servers. If you run Windows NT Server or Windows 2000 Server, you can use the DNS service included in those platforms to host your own DNS records. Otherwise, you can use a third-party DNS server application.

Next, you need to create or have your registrar create a host record in the DNS zone for your domain. The host record associates a host name with an IP address. For example, the host name of my mail server is mail.boyce.us. In addition to the host record, you also need an MX record. This mail exchanger record tells other mail servers what address to use to deliver mail to your domain. Again, you need to create the MX record on your own DNS server or have your registrar create it on its servers, depending on where your DNS service resides. Both records need to point to your public IP address.

Now it’s time to hook up your server. If yours is like most networks with a broadband connection, you have a small number of public IP addresses (perhaps only one), and all of your computers use private IP addresses. The MX record must reference the public IP address because that’s the only one the outside world can see. So, if your DSL router or cable modem is assigned the one public IP address, it needs to forward the incoming SMTP traffic to the private IP of your mail server.

Check the documentation for the unit and see if it supports one-to-one Network Address Translation (NAT). Many cable/DSL routers let you translate specific ports, so you would configure the unit to pass port 25 (SMTP) from the public IP to the private IP of your mail server. If you’re using your own DNS server, you’ll also need to translate port 53 to move DNS traffic to the private IP address of your DNS server (probably the same computer that’s handling mail). If anyone, including you, needs to be able to retrieve messages from the server outside of your network (such as from the Internet), you also need to translate port 110 (POP3) from the public IP to the private IP of the mail server.

If your cable/DSL router doesn’t support NAT, you’ll have to either replace it or obtain a second public IP address from your ISP. Then, set up the server with that IP address. How you hook the server into the network depends on the type of equipment you use, so check with your ISP if you aren’t sure.

The last step is to install the mail server software and start setting up and configuring accounts. You’ll also need to set up DNS and get that working if you’ll be providing your own DNS services. Then, you can start e-mailing to your heart’s content.

Subscribe to the Developer Insider Newsletter

From the hottest programming languages to commentary on the Linux OS, get the developer and open source news and tips you need to know. Delivered Tuesdays and Thursdays

Subscribe to the Developer Insider Newsletter

From the hottest programming languages to commentary on the Linux OS, get the developer and open source news and tips you need to know. Delivered Tuesdays and Thursdays