id="info"

Data Centers

Get IT Done: Troubleshooting ISA Server's server publishing rules

Determine the reasons for broken ISA Server publishing rules and learn how to fix them.


If you’re using ISA Server as your firewall, then you can use server publishing rules to allow Internet hosts to access services on your internal network. ISA Server’s server publishing rules provide the same kind of reverse network address translation (NAT) you get with the Routing and Remote Access NAT Service, but with a few extras. You can control access to a particular server publishing rule by allowing only computers contained in one or more Client Address Set. You can even create exceptions to allow or deny rules based on a Client Address Set.

After you create them, server publishing rules work without any further intervention on your part in the majority of cases. However, there are some specific problems that you might run into. You’ll end up tearing your hair out trying to determine what’s wrong if you’re not aware of these problems. In this Daily Drill Down, I’ll show you what you need to know to prevent unnecessary frustration.

Author’s note
For the purposes of this article, I’m not going to take you through the process of how to create server publishing rules. For more information about server publishing rules, see the Daily Feature “Publish network resources with ISA Server.”
Exchange server publishing also takes advantage of server publishing rules. In this article, I’ll focus on problems that span the range of server publishing rules. In a future article, I’ll cover the special problems involved with publishing Exchange servers.


What can possibly go wrong?
Normally, server publishing rules work fine after you set them up. However, because they’re so complex, you could run into some type of problem while implementing them. Some of the more common troubleshooting issues you’ll encounter with ISA Server’s server publishing rules include:
  • Socket pooling issues.
  • Problems with publishing the same service multiple times.
  • Problems with publishing complex protocols.
  • Binding a service to a particular IP address on the external interface of the ISA Server.

Problems with socket pooling
Microsoft Internet Information Server 5.0 uses a feature known as socket pooling that allows IIS to use resources more efficiently. Socket pooling allows IIS services, such as the Web service, mail service, and news service, to bind to all interfaces. Allowing IIS to listen on all ports usually doesn’t cause a problem on an intranet computer, but it can be a big problem if the server is also serving as a firewall using ISA Server.

One of the most common scenarios for this problem is when you want to publish a mail server on your internal network. Initially, you don’t receive any error notifications when you create the SMTP server publishing rule. However, when you try to use SMTP, you discover that the SMTP server publishing rule doesn’t work. The most likely reason for this is that the SMTP service (smtpsvc) is enabled on the ISA Server. Rather than attempting to address the published mail server, the SMTP request gets forwarded to the ISA Server.

You can fix such a problem two ways. You can either disable the SMTP service on the ISA Server or disable socket pooling for the SMTP service.

You don't have to do anything fancy to disable SMTP on the server. Just disable the SMTP services in the Services Microsoft Management Console (MMC) as you would with any other Windows service. You might want to continue running the SMTP service on the ISA Server so that the machine can also act as an SMTP message screener. To continue running the IIS SMTP service and avoid the problem, you’ll need to disable socket pooling.

To disable socket pooling for the SMTP service on the ISA Server, you’ll need the Mdutil.exe utility. The Mdutil.exe utility isn’t installed with Windows 2000, but it is on the Windows 2000 Server CD-ROM. Do a search for Mdutil and use the Expand utility to decompress the file onto your hard disk.

Copy the Mdutil.exe executable to the \Inetpub\Adminscripts folder. Next, open a command prompt window and stop the SMTP service by typing net stop smtpsvc. After the service stops, change to the \Inetpubl\Adminscripts directory. Type the following command:
mdutil set –path smtpsvc/1 –value 1 –dtype 1 –prop 1029 –attrib 1

Next, go to the Internet Information Services console, right-click on the SMTP service, and click Properties. Change the listening address to an IP address on the internal interface of the ISA Server. Finally, restart the SMTP service with the net start smtpsvc command from your server’s console prompt.

If you have multiple SMTP servers, you will need to change the value for the smtpsvc/1 switch. The “1” indicates the number of the virtual SMTP server. The first virtual SMTP server in the IIS console is number 1, the second is number 2, and so on. Make sure you disable socket pooling for all virtual SMTP servers.

The same principles apply if you need to disable socket pooling for the Microsoft NNTP service. The only difference is that you specify the NNTP service instead of the SMTP service. For example, mdutil set –path nntpsvc/1 –value 1 –dtype 1 –prop 1029 –attrib 1disables socket pooling for the NNTP service. As with the SMTP service, make sure you disable socket pooling for each virtual NNTP server.

You have to use a different procedure if you want to disable socket pooling for the Web service or the FTP service. This method takes advantage of an Administrative script provided with IIS. To disable socket pooling for the FTP service, open a command prompt and change directories to the \Inetpub\Adminscripts\ folder. Type net stop msftpsvc and press [Enter]. Next, type:
cscript adsutil.vbs set msftpsvc/disablesocketpooling true

and then press [Enter]. Restart the FTP service with the net start msftpsvc command.

The same script works with the Web Proxy service. Just replace the msftpsvc switch with w3svc.

Publishing multiple services on the same IP address
Most organizations that publish mail servers will publish SMTP relays rather than the corporate mail server itself. This is a good security practice since it prevents Internet users and servers from directly communicating with the corporate mail server. Another advantage of using multiple SMTP relays is that you can take one SMTP relay down and mail will still be sent to the other relay as long as you have configured the appropriate MX records.

Problems can occur with ISA Server’s server publishing rules and SMTP relay servers because you can’t publish both of the SMTP relay servers if you have a single IP address bound to the external interface of the ISA Server. You can bind a particular port only once per IP address. Therefore, if SMTP Relay no. 1 uses TCP port 25 for its server publishing rule, the port will not be available when you want to publish SMTP Relay no. 2. If you try to do this, you’ll see the error message that appears in Figure A.

Figure A
You’ll see this error when using the same port twice on the external interface of the ISA Server.


You can solve this problem by binding a second IP address on the external interface of the ISA Server. Although you can use Web publishing rules to publish multiple Web servers on the internal network with only a single IP address on the external interface of the ISA Server, server publishing rules require that you have a separate IP address for each instance of a published service (protocol).

Publishing complex protocols
In an ISA Server world, a complex protocol is one that requires secondary connections. The classic example of a protocol that requires secondary connections is port (standard) mode FTP. When your FTP client is configured as a port mode FTP client, it sends a request over the FTP control channel (TCP port 21). The FTP server then needs to establish a new inbound connection to a high port from its own TCP port 20 to create the data channel. The new inbound connection represents a secondary connection. If the firewall in front of the FTP client or server isn’t aware of what’s taking place between the FTP client and server, the FTP transfer will fail.

There are two ways ISA Server can deal with publishing complex protocols. It can either use application filters or it can use a combination of the firewall client and wspcfg.ini files. Application filters are the best way to deal with complex protocol issues. They are the only way to allow SecureNAT clients to use complex protocols.

Without an application filter, the SecureNAT client cannot take advantage of the secondary connections required by complex protocols. Since almost all of your published servers will be configured as SecureNAT clients, you must use application filters to support publishing complex protocols for these clients.

Fortunately, ISA Server comes with some application filters you can use right out of the box. The FTP access filter and the RPC filter allow you to publish FTP and Exchange servers on your internal network without needing to install the firewall client on these servers to support the complex protocols. The FTP and RPC filters make it easy to publish these servers using a simple server publishing rule.

You can also use the firewall client and a wspcfg.ini file to publish servers that require complex protocols. This was the only Server Publishing method available to Proxy 2.0 administrators. The configuration of wspcfg.ini files for publishing complex protocols is a complicated and often frustrating process. However, if you do not have C++ programmers on your staff and you need help configuring wspcfg.ini files, there are two excellent resources you can use. The first place to start is Jim Harrison’s excellent article on the Firewall client at ISAserver.org’s Firewall client article by Jim Harrison. After reading that article, head on over to the ProxyFAQ to see if it has the information you need.

Binding an IP address on the ISA Server
When you publish an internal network server using a server publishing rule, ISA Server configures its default configuration so that any outbound messages from that server will have a source IP address set to be the primary IP address bound to the external interface of the ISA Server. This can be a problem if destination hosts authenticate a computer using a reverse DNS lookup.

For example, suppose you publish two SMTP servers on your internal network. One SMTP server uses the IP address 222.222.222.1 on the external interface of the ISA Server, and the second SMTP server uses 222.222.222.2. You have configured two Host (A) and MX records that map to these two IP addresses. If 222.222.222.1 is the primary IP address on the external interface of the ISA Server, all outbound packets from all publishing servers on the internal network will show a source IP address of 222.222.222.1. If a destination SMTP server does a reverse lookup on messages sent from the SMTP server that was published on 222.222.222.2, it will find that the source IP address in the packet and the reverse DNS lookup result don’t match and therefore may reject the message as a possible spoof.

There is no configuration option on the ISA Server that will allow you to bind an internal server to a particular IP address on the external interface of the ISA Server. Your only option is to use the firewall client on the publishing SMTP server and a wspcfg.ini file to accomplish this task.

For example, suppose you wanted to publish the IIS 5.0 SMTP service. The IIS SMTP service runs as a process within the inetinfo.exe application. You want to bind the address 222.222.222.2 on the external interface of the ISA Server to this particular SMTP server. The wspcfg.ini would look like this:
[inetinfo]
ProxyBindIP=25:222.222.222.2  
ServerBindTcpPorts=25
Persistent=1
KillOldSession=1


Install the Firewall client on the SMTP server and place wspcfg.ini file in the same directory as the inetinfo.exe application. Make sure that socket pooling is disabled on the ISA Server and that another server publishing rule isn’t using the same port on the same IP address.

Some rules aren’t meant to be broken
ISA Server’s server publishing rules make it easy for you to make servers on your private network available to users on the Internet. You will find that, in the majority of cases, your server publishing rules will work right out of the box. However, there will be times when they don’t. Once you know the most common reasons for broken server publishing rules, you can quickly figure out a way to fix them.

Editor's Picks