Data Centers

Lock IT Down: Avoid these ISA Server and Proxy Server configurations

Configure ISA Server and Proxy Server properly

Both Proxy Server and ISA Server do a really good job of defending your network from the outside world—if you've taken the time to configure them properly. Like all Windows Server products, both can be a little difficult to configure. There are several configurations that will cause the products to malfunction, or, even worse, that can seriously compromise your network's security. Here are the configurations that network administrators commonly use and why they can cause you problems.

Two sides of the same coin
The configurations that I'm describing in this Daily Drill Down are for ISA Server. However, most of them also apply to Microsoft's Proxy Server 2.0. If you aren't familiar with Proxy Server 2.0, it was the predecessor to ISA Server and did many of the same things. Therefore, configurations that can cause a problem with ISA Server can also be problematic for Proxy Server.

Most often both ISA Server and Proxy Server are loaded onto a dedicated machine containing two NIC cards. One NIC is attached to the local network, while the other NIC is attached to the Internet. It's then the ISA Server's job to filter the traffic that's flowing between the local network and the Internet. As I discuss the various configurations in the sections below, I will assume that this basic implementation is being used unless otherwise noted.

Inbound/outbound with a DMZ
One of the riskiest configurations is also one of the most common. This configuration involves using ISA Server to regulate inbound and outbound traffic, as well as to regulate traffic flowing between the Internet and a DMZ. In case you aren't familiar with DMZs, a DMZ is a "demilitarized zone." The idea is that servers within the DMZ are completely unprotected by the firewall and are totally accessible from the Internet.

Most of the time when a DMZ is used, people place Web servers or mail servers into it, so that the company's Web site or SMTP e-mail is accessible to the outside world. As an added security measure, I have seen some companies use a third NIC in an ISA Server. This NIC is specifically used to route traffic between the Internet and the DMZ.

So what's so bad about this configuration? There are two things really. First, if you implement this configuration from within Microsoft Proxy Server, it requires you to enable routing on the server. This is a huge security risk because the server's routing mechanism could potentially be exploited and used to gain access to your private network. This isn't really a problem with ISA Server, since ISA handles the routing process differently from Proxy Server.

The other problem with using ISA Server in conjunction with a DMZ isn't so much an ISA configuration issue as it is that DMZs are inherently insecure. There are times when it may be appropriate to use a DMZ, but I would personally implement a DMZ only as a last resort. Instead, I would recommend placing a server that will be publicly accessible into its own private network, just as you would if you were creating a DMZ. However, instead of simply opening the server up to the public, I recommend configuring the authoritative DNS for that server to point to the DNS entry to the ISA Server instead of your publicly accessible server. You can then configure ISA to use an access policy to forward requests to the publicly accessible server.

This means that the server is never directly exposed to the public. Instead, the ISA Server is the only machine that ever talks to the publicly accessible server. The ISA Server sends requests to the publicly accessible server on the user's behalf. This does two things for you.

First, it hides the server from public view. By doing so, you hide things like the server's IP address and computer name. Second, you can filter out undesirable traffic. For example, if the publicly accessible server is a Web server, then you probably want everyone in the world to be able to access it. However, you only want this access to occur through TCP port 80. Using ISA Server, you can block all other TCP and UDP ports, reducing the chance that someone will hack your Web server.

Public isolation
No machine, whether it's on your private network or on the Internet, should be able to access your publicly accessible systems without passing through the ISA Server. Although this statement is fairly obvious, there are two related actions that I sometimes see taken that tend to jeopardize security.

First, I sometimes see organizations use separate subnets for the publicly accessible systems and the private network, but use the same hub or switch to connect public and private systems. This is an extremely bad idea. If there is a hardware path between the two networks (other than through the ISA Server), then it's a matter of time before someone figures out how to exploit that connection.

The other blunder that I sometimes see involves hardware upgrades. Twice, I have seen companies get a new Web server and then move the old Web server to their private network for use as a file or application server. While there's certainly nothing wrong with repurposing hardware, there is something wrong with moving a previously publicly accessible system to a private network. Once a system has been made public, that machine should be considered untrustworthy.

Although you probably do your best to maintain the integrity of your publicly accessible machines, the fact remains that those machines have had contact with untrusted (unauthenticated) people through the Internet. While you would like to think that no one used your server for anything other than its intended purpose, it's possible that someone could have planted a Trojan or a spyware module onto the server. Therefore, once a server has had contact with untrusted users, the server should be considered tainted. I personally would never move an old server from a public network to a private network without performing a secure format on it first and then reinstalling Windows.

The Local Address Table (LAT)
Another common ISA Server problem that's more directly related to the server's configuration is the way that the Local Address Table (LAT) is configured. Normally, the LAT should contain the IP addresses of all of the machines on your internal network. This allows the ISA Server to distinguish between machines on your private network and machines accessing your network through the Internet.

The problem I occasionally see is that people will place the IP address of both of the ISA Server's NIC cards into the LAT. This is a huge mistake because, in certain environments, it will allow Internet-based users to browse your private network. In the case of Proxy Server, if IP forwarding is enabled on the server (as would be the case with a DMZ), packets could be forwarded to your private network. This is also possible if an Internet user has a modified version of the Microsoft Proxy Client installed.

Default gateway
One of the lesser known ISA and Proxy Server configuration issues involves the machine's TCP/IP configuration. As I explained earlier, a typical ISA Server contains two NIC cards. One is attached to the Internet, while the other is attached to the private network. Normally, machines on your network that access the Internet through the ISA Server either won't have a default gateway configured, or the default gateway will be the ISA Server's IP address on the private network. It's very important that the ISA Server itself not be configured in this manner, though.

If the ISA Server's NIC that's connected to the Internet has a default gateway matching the ISA Server's private IP address (the address that the ISA clients on your private network are using for a default gateway), then it may become possible for Internet users to forward packets to your private network. If you must use a default gateway for your ISA Server's Internet NIC, then use the card's own IP address as the gateway address.

Excessive services
A couple of months ago, I did some consulting work for a company that had decided to save some money by running multiple Microsoft Server products on a single box. While this is not at all uncommon for smaller companies to do, the problem was that in addition to running Exchange 2000 and SQL, the server was also running ISA Server. The box was acting as both a firewall and as an application server.

As you can imagine, this is just a bad idea. You never want your mission-critical applications loaded onto a server that's directly connected to the Internet. The company had definitely managed to save a few bucks on server licensing, but had risked their network's overall security in doing so.

A less dramatic example of a poorly configured ISA Server is a server that's running lots of different services. If a Windows 2000 Server has all of the appropriate service packs and hot fixes installed, there are no huge security holes in any one service. However, there are lots of documented techniques in which minor bugs or even features in a service can be exploited in conjunction with bugs or features in other services, resulting in a massive security breach. The point is that any of the services by themselves were relatively harmless, but when used together they can compromise security.

My advice is to disable any service that isn't absolutely necessary on the ISA Server. ISA should be the server's sole application and anything that isn't specifically required for ISA needs to be disabled. Remember that your ISA server absolutely must be the most secure server on your network.

Bad filter rules
So far, I've just been covering the basic do's and don'ts of ISA Server. Generally speaking, the techniques I've discussed so far will take care of most of the ISA-related security problems. There is one more area that needs to be examined though: Unless you have the appropriate filter rules in place, your ISA Server is useless.

One of the great things about ISA Server is that out of the box it already has a relatively secure configuration. You can make that configuration even more secure by tweaking the filter rules. However, these tweaks can also cause problems. One such problem involves confusing the ISA Server. I once saw a situation in which someone somehow managed to configure an ISA Server with contradictory filter rules.

While ISA Server does have an algorithm for dealing with contradictory rules, in doing so it ignores one of the two contradictory rules. This means that either a service you want to support will be blocked, or traffic you want to block will be allowed to flow onto your private network.

My advice is to set up an ISA Server in a test environment prior to implementing it as your company's main firewall. In doing so, you will have the opportunity to experiment with various filter rules. You may also use some of the various hacker tools to experiment with trying to hack your ISA Server before you use the server to guard anything important.

Another common problem involves packet filtering. I strongly recommend that you filter packets on all ports that aren't specifically needed. Be especially sure to block ports 135, 137, 138, and 139. These are the ports used by Window's networking services. If you forget to block these ports, you're opening your network up to the most devastating types of attacks. Worse yet, attacks against your network won't just come from professional hackers.

I know a couple of hacker wannabes that can't hack anything else, but that could hack a system if one of these ports were left open. It's so easy to hack a system through these ports because of the way that Windows uses them and because so many hacker tools have been designed to exploit these ports through a nice GUI interface that anyone can figure out.

Although not really an issue in ISA, a common misconfiguration in Proxy Server involves enabling the option to analyze UDP packets. On the surface, analyzing inbound UDP packets sounds like a good thing to do. However, it's possible to flood a Proxy Server with more UDP packets than it can handle if this option is enabled. This makes it very easy to launch a UDP-based denial of service attack against the Proxy Server.

Avoid at all costs
Keep the guidelines in mind to avoid troublesome configurations:
  • Don't place publicly accessible servers on the same hub as is used for your private network.
  • Never move a publicly accessible server to your private network without reformatting it first.
  • Avoid using a DMZ unless you have no other choice.
  • Configure ISA Server to filter any packets that you don't specifically need.
  • Make absolutely sure to block ports 135, 137, 138, and 139.
  • Avoid using complex filtering rules that could become contradictory and confuse the server.
  • Always test your ISA Server's filtering rules in a lab before relying on them to defend your network.
  • Never run anything other than ISA Server on the server hardware, and disable any services that aren't specifically required by ISA Server or Windows.
  • Make sure that your external NIC's IP address doesn't appear in ISA Server's LAT.
  • Double-check your ISA Server's TCP/IP configuration to make sure that the correct default gateway is being used.

Editor's Picks

Free Newsletters, In your Inbox