Software

TechRepublic Tutorial: Secure SMTP traffic with ISA Server

With some help from ISA Server, you can quickly secure SMTP and your Exchange server.


If your Exchange Server is connected to the Internet, allowing users to send and receive Internet e-mail, then you’re probably using the SMTP protocol. As you’re no doubt aware, the Internet tends to be a very hostile environment, and it is therefore critical to safeguard your Exchange Servers and the SMTP protocol that they use. One way of doing this is by using Microsoft’s ISA Server as a go-between.

Author’s Note
Although this article discusses ways of securing the SMTP traffic flowing between the Internet and your Exchange Server, much of the information in this article is applicable even if your organization doesn’t have an Exchange Server. Remember that even if your users simply use Microsoft Outlook to connect directly to an Internet-based mail server that’s provided by your ISP, SMTP is still being used.

The ISA Server Message Screener
In almost all of the real-world ISA Server implementations that I have seen, ISA Server was installed as an integrated caching firewall server. However, ISA Server does have some lesser-known add-on components that come with it. One such component is the Message Screener. Enabling the Message Screener is a key step toward securing SMTP traffic.

You can install the Message Screener by opening the Control Panel and double-clicking the Add/Remove Programs icon. When you do, you’ll see a list of all of the currently installed applications. Select ISA Server from the list and click the Change button. You’ll be prompted to insert your ISA Server CD and will then be presented with a dialog box that gives you the option of changing, reinstalling, or uninstalling ISA Server. Click the Add/Remove button to reveal a list of the ISA Server components.

At this point, select the Add In Services option and click Change Option. Select the Message Screener check box and click OK, followed by Continue. When you do, ISA Server will stop the necessary services and install the Message Screener. Although you may install the Message Screener at any time, your ISA Server must have an external IP address configured prior to being able to install the SMTP filtering components. On my test server, I had some trouble getting ISA Server to recognize my external IP address, which was configured on my server’s second NIC. I was able to solve the problem by making sure that my external IP address didn’t appear on the ISA Server’s local address table and then rebooting the server.

Configuring Message Screener
After you have installed the Message Screener, you must configure it. Before doing so, you must ensure that the ISA Server is connected to an external IP address by which SMTP traffic can flow between your network and the Internet. To do so, open the ISA Server Management console. Once the console is open, navigate through the console tree to Internet Security And Acceleration Server | Servers And Arrays | your server | Publishing | Server Publishing Rules. Now, right-click on the Server Publishing Rules container and select Secure Mail Server. This will launch the Mail Server Security Wizard.

Click Next to bypass the wizard’s welcome screen. You’ll see a screen asking you which mail services you would like to publish to your external users. At the very least, you’ll probably want to publish incoming SMTP and outgoing SMTP. If your Exchange Server is SSL-enabled, you may also want to publish the SSL implementations of the incoming and outgoing SMTP services. You’ll also want to use the Incoming Microsoft Exchange/Outlook option, especially if your users are receiving messages directly within Microsoft Outlook, rather than retrieving them through Outlook from an Exchange Server. You can see an example of this in Figure A.

Figure A
Choose the mail services that are appropriate for your organization.


The next screen you'll encounter asks for the external IP address of the ISA Server. Normally, you would probably know this address, but if you're working on a client’s server and don’t want to take the time to look up the address, you can use the Browse option to quickly locate the external IP address.

The next screen will ask you for the IP address of your internal mail server. Normally, you would just enter the mail server’s IP address into the space provided. However, there are situations in which Exchange Server and ISA Server might be running on the same computer. In this case, you would use the On Local Host option rather than entering an IP address.

When you click Next, you’ll see a summary screen that explains which services are being used and also lists your server’s public and private IP addresses. Review this information, and if everything looks good, click Finish.

Additional configuration requirements
So far, you have configured ISA Server to act as an SMTP Server. When someone sends an SMTP message, he or she will actually send it to the ISA Server rather than to the Exchange Server. SMTP will then filter the message and relay it to the Exchange Server, if appropriate. In order to achieve this functionality, there are a few housekeeping chores that you must complete prior to continuing.

First, you must point your SMTP-related DNS entries to the ISA Server rather than to your Exchange Server.

The next thing that you must do is check to see how port 25 is configured within your existing firewall structure, since port 25 is used for SMTP traffic. Normally, you would just make sure that port 25 is open. However, if your ISA Server is behind a traditional firewall, you should set your real firewall to forward traffic on port 25 to the ISA Server.

As you’ve already seen, you have a lot of options with mail traffic. If you plan on dealing with any sort of authentication over port 25, you should implement SSL for SMTP. Also, if your ISA server is a member of an ISA array, you cannot publish outbound ISA traffic.

You must also make sure that Windows 2000 Service Pack 2 or higher and ISA Server Service Pack 1 or higher are installed on your ISA Server. You might also look into Service Feature Packs available for ISA Server that you can use to further increase security.

Filtering SMTP traffic
Now that ISA Server has been configured to accept SMTP traffic, it’s time to configure SMTP filtering for the server. To do so, navigate through the ISA Server Management console to Internet Security And Acceleration Server | Servers And Arrays | your server | Extensions | Application Filters.

When you select the Application Filters container, you should see a list of various filters appear in the column to the right, as shown in Figure B. Right-click the SMTP Filter and select the Enable command from the resulting shortcut menu if the SMTP filter isn’t already enabled. Once the filter has been enabled, right-click the SMTP Filter once again, but this time select the Properties command from the shortcut menu. This will display the SMTP Filter’s properties sheet.

Figure B
ISA Server includes basic filters that you can customize as needed.


As you look over the properties sheet, you’ll notice that there isn’t really anything on the General tab other than an option to enable or to disable the filter. The real action is on the other tabs.

The Attachments tab
The Attachments tab is set up to allow you to create a series of filter rules pertaining to SMTP message attachments. You can use the Add button at the bottom of this tab to create a new rule. Later, you can use the Remove button to delete it or the Edit button to modify a rule that was previously created. If you click the Add button, a dialog box will appear. This dialog box gives you three choices:
  • Filter attachments based on filename
  • Filter attachments based on file extension
  • Filter attachments based on file size

Select the appropriate filter type by clicking its radio button. You can then specify the filter by entering information in the fields next to each selection. The Action dropdown list box describes the action the filter should take when your filter is triggered. ISA Server can delete the message, hold the message, or send the message to a specific e-mail address.

Remember that the dialog box is designed to set up a single rule, not to act as a catchall. If you need to screen multiple types of attachments or take different types of actions when attachments arrive, you’ll want to set up multiple rules.

Like most other filters, the Mail Attachment Rule dialog box allows you to set up a filter rule and a filter action. The filter rule can be based on an attachment name, attachment extension, or attachment size limit. The action can consist of deleting the message containing the attachment, holding the message, or forwarding the message to someone specific.

In the real world, filtering based on a specific attachment name is ineffective unless you have one specific attachment that keeps rearing its ugly head. I have seen organizations that do use this filtering system, though. For example, if the attachment is an EXE file, you might delete it automatically since it could be a virus. Likewise, if the attachment is a JPG file, you might forward it to your own mailbox so that you can screen for porn or for other offensive material.

On my own network, I don’t filter based on attachment name or extension. I simply rely on my anti-virus software to catch any malicious attachments. My favorite choice for anti-virus protection is ViRobot. ViRobot does an excellent job of catching these malicious attachments and removing them before they can do any damage.

I do, however, filter messages based on size. When I set up my network, I didn’t want to set a limit on my Exchange mailboxes because I sometimes receive perfectly legitimate attachments that are up to 200 MB in size. The fact that some of these attachments are executable programs is another reason why I don’t filter based on file type. However, I don’t want someone sending me some huge file that is going to run my server out of disk space. Therefore, I have ISA Server set to block any message containing an attachment over 1 GB in size.

The Users/Domains tab
The Users/Domains tab tends to be a bit more self-explanatory than the Attachments tab. As you can see in Figure C, the Users/Domains tab is set up to allow you to block specific individuals or domains from sending you SMTP mail. Simply input the sender’s name or the domain name into the space provided and click the Add button. The sender or domain will be added to the list of blocked senders or domains to the right. You can remove a sender or domain from the list by selecting the appropriate name and clicking the Remove button.

Figure C
The Users/Domains tab allows you to block specific users or domains from sending SMTP mail to your Exchange users.


I have seen some people use this tab as a way of blocking known spammers. Generally, though, this technique is ineffective. The reason is that spammers often spoof e-mail addresses and domains, or use a different e-mail address each time a SPAM is sent. A more practical use of this filter is to block known individuals or domains. For example, there is one specific user I know of who is infected with the Melissa virus, but has yet to do anything about it. Until I implemented a filter to block mail from this user, I received dozens of e-mail messages from him every day.

A few years ago, I ran into another case that called for filtering by user name. At the time, I was a CIO for a chain of hospitals. One of the women at one of the hospitals was going through a nasty divorce. Her soon-to-be ex-husband was constantly harassing her through e-mail, and I was able to block his messages by setting up a filter similar to this one.

The Keywords tab
The Keywords tab allows you to create a series of rules to block messages containing specific keywords. To create a keyword rule, click Add and you'll see the dialog box shown in Figure D.

Figure D
The Mail Keyword Rule dialog box allows ISA to take action against messages containing specified keywords.


This dialog box is simple to fill in. You must enter the keyword of choice and then choose whether you want to look for the keyword in the message header, body, or both. You may then choose to delete, hold, or forward messages containing the specified keywords.

The one thing that you need to know about entering keywords is that you have to block the exact word, which is something that many SPAM senders take advantage of. For example, you might block the keyword “porn.” But many spammers will place other characters between the letters of such a keyword, such as P$o$r$n, p o r n, and po rn. From an ISA Server standpoint, each of these are different keywords, and must be entered separately.

Furthermore, unless you have ISA Server service pack 1 installed, don’t try to insert a space into a keyword, or you’ll cause some problems. Generally speaking, you can configure the keyword filter to block a lot of SPAM, but it’s going to be tough to catch all of it.

The SMTP Commands tab
The SMTP Commands tab, shown in Figure E, is the tab to which you’ll probably have to do the least configuration, but it is, in my opinion, one of the most useful tabs. The idea behind this tab is that it filters SMTP commands that are beyond a specific length. For example, suppose for a moment that someone were to send an SMTP message with an invalid header length. Some e-mail client programs automatically check header lengths, but many do not. If such a client received an SMTP message with an invalid header, it could crash the program, corrupt the database, or do other types of damage.

Figure E
The SMTP Commands tab checks for valid SMTP command lengths.


As you can see in the figure, the SMTP Commands tab is preset to the appropriate values. Normally, you should never have to touch these settings. However, you do have the option of adding additional commands, modifying existing command values, or removing a command filter completely, by using the Add, Remove, and Edit buttons.

Safe and sound
When you’re using Exchange to power your Internet-based e-mail, you must learn to deal with SMTP and all of the security problems it can introduce. Fortunately, with some help from ISA Server, you can quickly secure SMTP and your Exchange server.

Editor's Picks

Free Newsletters, In your Inbox