Obviously there are many different things that you can do to secure and Exchange Server. Some of the security techniques are naturally more important than others. Here is a list of tips you can use to secure your Exchange server.
Probably one of the most important, if not the most important, things that you can do to secure your Exchange Server is to implement relay restrictions. It isn't like hackers will be a little break into your server if you don't implement relay restrictions, but relay restrictions are still extremely important. Relay restrictions keep spammers from having a field day with your server. If relay restrictions are not implemented, a mail server is said to be an open relay. What this means is that anyone, including spammers, can relay mail through your server. Spammers commonly used this technique to obscure their IP address. If messages have been relayed through your Exchange Server, the messages will appear to have come from your server's IP address rather than the IP address of the spammer's mail server.
This causes a couple of problems. First, it will only be a matter of time before your Exchange Server's IP addresses blacklisted. Your server's IP address will most likely become blacklisted because it is being used to send spam. Some organizations will blacklist any mail server with an open relay even if that server has never been used to send spam.
Another problem with having an open relay is the impact that it may ultimately have on your server's system resources. Spammers do not send messages in small quantities. As thousands of spam messages are relayed through your server, resources such as Internet bandwidth, CPU time, and disk space are all being used to facilitate the transport of spam. It is fairly common for so many resources to be used by spammers relaying mail through a server, that the remaining resources are insufficient to meet the organization's operational needs.
The good news is that implementing relay restrictions is extremely easy to do. To do so, open the Exchange System Manager and navigate through the console tree to Administrative Groups | your administrative group | Servers | your server | Protocols | SMTP | Default SMTP Virtual Server. Now, right-click on the Default SMTP Virtual Server and select the Properties command from the resulting shortcut menu. When you do, you will see the Default SMTP Virtual Server Properties sheet. Select the properties sheet's Access tab and click the Relay button. You will now see the Relay Restrictions dialog box.
To close an open relay, choose the Only The List Below option. If you have users who need to relay messages through your server, you can create an exception list by using the Add button. This dialog box also contains a check box that you can use to allow any computer which successfully authenticates to relay regardless of the contents of the exception list. Click OK twice and the open relay is closed. Keep in mind that if you have additional SMTP virtual servers, you'll have to close the open relay for each virtual server individually.
When it comes to protecting things like file servers and domain controllers against viruses, pretty much any server grade antivirus software will do. When you throw in an Exchange Server though, the antivirus software requirements change drastically. When your organization hosts its own mail server, files are routinely moved in and out of the organization through that mail server. As such, it is necessary to protect against viruses on many different levels.
There are actually two different types of antivirus software that need to be run on Exchange servers. The first type is file level antivirus software. This is the same type of antivirus software that you would use to protect any other type of server. Its job is to monitor the server's file system and remove any viruses that it may find.
Next, the server needs some antivirus software that is specifically designed for Exchange Server. The reason for this is because e-mail viruses do not exist on an Exchange Server in the form of standalone files. Instead, they are stored in the Exchange mailbox store along with all the other messages. You therefore need an antivirus application that knows how to read an Exchange database, and how to remove an infected file from an Exchange database without corrupting the database and the process.
The next level of protection that you'll need is workstation level antivirus software. There are two main things that you should look for in workstation antivirus software. First, the software needs to be designed so that it integrates itself into Microsoft Outlook. That way the software can scan messages as they are opened.
Second recommendation that I would make regarding workstation antivirus software, is that you should use something different than what is running on your Exchange Server. For example, if you are running Norton Antivirus on your Exchange Server, then you might consider running software from McAfee or Trend Micro on your workstations.
The reason for doing this is that when new viruses are discovered, you never know which antivirus company is going to publish a signature for the new virus first. Imagine for example that you were running Norton Antivirus at both the server and the workstation level. Now imagine that a new e-mail virus was released and that Symantec had not yet published a signature for the virus. If someone were to e-mail a copy of the virus to someone in your organization, there would be no preventing an infection because you have no signatures to defend against the virus.
Now suppose that instead of running Norton Antivirus on your workstations, you were running Trend Micro's PC-cillin. If the same virus were e-mailed to someone in your organization, the virus would not be eradicated at the Exchange Server level because Norton Antivirus does not yet have a signature for the virus. However, there is a possibility that the virus could be stopped at the workstation level. Norton Antivirus doesn't have a signature yet, but PC-cillin might. Of course the next time around Symantec might beat Trend Micro in publishing a signature. The point is that if you use multiple antivirus vendors, you double your chances of having a signature in place when a virus arrives.
There is one more type of antivirus software that I would recommend having in place. I would recommend using a Gateway level antivirus product. A Gateway level antivirus product stops viruses as they flow in or out of your organization. As I'm sure you know, most of the e-mail viruses that have come out are designed to replicate themselves by sending a copy of themselves to all of the victim's contacts.
If one of your users were to somehow become infected by an e-mail virus, that virus would probably try to e-mail copies of itself to people outside of your organization. Your gateway level antivirus software could delete the infected messages as they are leaving your organization.
RPC over HTTP
There are many ways of configuring an Exchange Server to be accessible to remote clients. Probably the most common methods of facilitating remote connectivity are too implement Outlook Web Access or to implement a VPN. If your company allows remote users to connect to the corporate network over a VPN for the sole purpose of checking their mail, you might consider implementing RPC over HTTP as an alternative.
RPC over HTTP has many of the same benefits as a VPN connection. Remote users are able to use Microsoft Outlook to check their mail just as they could if they were physically attached to the company's local area network. In fact, an RPC over HTTP connection will work even if the Exchange Server and the remote user are both behind firewalls.
The reason why I recommend taking a look at RPC over HTTP is because unlike a VPN connection, an RPC over HTTP connection limits the scope of what resources can be accessed remotely. For example, if a user is attaching to the corporate network using a VPN connection, that user will have access to the same resources that they would if they were physically attached to the company's local area network. That's great assuming that the remote user actually needs access to all of those resources. If the remote user is only connecting to check their mail though, then exposing other resources over remote connection is a bad idea. You never know when a user's password could be stolen. In such a situation, a VPN connection is like an open door for the password thief.
On the other hand, if you are only using a RPC over HTTP connection for remote users then fewer resources are exposed to remote connections. If someone stole a user's password, they might be able to get access to the user's e-mail, calendar, and contacts, but your file servers would not be compromised.
Protecting front-end servers
The generally accepted way of implementing a front end / backend configuration involves placing an ISA Server in front of the front end server. The idea behind this configuration is that remote clients never interact directly with the front end server. Instead, the ISA Server is provided with a copy of the front end server's certificate, which allows it to impersonate the front end Exchange Server. Remote clients never actually communicate with the front end Exchange Server. Instead, remote clients communicate with the ISA server. The ISA server acts as a proxy server and forwards HTTP requests to the front end Exchange Server.
Part of the ISA server's job is to act as an application firewall for Exchange Server. What this means is that ISA server knows what types of communications are considered normal for an Exchange Server environment. It is therefore able to filter out abnormal and potentially malicious packets.
The merits of using an ISA server are sometimes debated though. The reason why this is a debated topic is that ISA Server is a software based firewall sitting on top of a Windows operating system. Some people feel that it is therefore vulnerable to the same types of attacks that any other Windows server would be.
My take on this issue is that ISA Server should be considered an essential part of a front end / back end configuration. ISA Server is not a generic firewall. It was developed by Microsoft with Exchange in mind. It contains lots of Exchange specific filtering rules that will help to keep your Exchange Server secure. At the same time though, I believe that the fact that ISA Server rides on top of a Windows operating system does pose a security threat. In my opinion, the best way to counter this threat is to place a hardware-based firewall at your network's perimeter and then have your hardware firewall forward the inbound HTTP requests to an ISA Server located behind it.
Keep Exchange Server up-to-date
One of the most important things that you can do to secure your Exchange Server is to keep the server up-to-date. Microsoft routinely releases patches that correct bugs and known security vulnerabilities in both Exchange Server and in the Windows operating system.
I know a lot of administrators who are reluctant to apply patches as soon as they become available. The reasoning behind this is that if a patch happens to be buggy, it could affect a server's stability. This is a perfectly valid argument. On the flip side though you also have to take into account what the likelihood is of a non-patched vulnerability being exploited.
I recently saw a statistic claiming that on average it takes four days from the time of vulnerability is made public until an exploit is designed against a vulnerability. Â The point is that when Microsoft releases a patch for a known vulnerability, they are basically telling the world that there is a security problem with their software. Hackers know that not everyone keeps their systems up-to-date. As such, many hackers actively search for unpatched systems so that they can take advantage of known vulnerabilities.
As you can see, this poses a no-win situation for administrators. On one hand, you don't want to have to explain to your boss why the network got hacked when there was a patch available that would have prevented the hack. On the other hand, you don't want to blindly apply patches and end up crashing your network because of patch was buggy either.
My recommendation is to set up a patch test lab. The idea is that you can create an isolated environment that closely mimics your production environment, and use it to test patches. Granted, having duplicate networks can cost big bucks. The patch test lab does not have to be an exact replica of your production network though. For example, you can keep the costs down by using PCs instead of servers. The important thing to remember is that the machines in the patch test lab should be used configured similarly to their real-world counterparts. That way you can get a good idea of how your production machines are going to respond once a patch is applied.