When working with ISA Server configured to run a VPN, troubleshooting the VPN problems can be one of the most difficult tasks you’ll encounter. The ISA Server’s VPN Wizards do most of the work when creating the VPN server or VPN gateway, but they can’t help you resolve VPN problems. By learning about the problems you might face and how to fix them, however, you’ll be better prepared to face the challenge should problems arise. In this Daily Drill Down, I’ll show you how to identify and troubleshoot problems with ISA Server VPN connections.

What kind of problems can I expect?
Most ISA Server VPN problems are related to VPN server or VPN client configuration and not to the actual ISA Server setup. This is because the ISA Server software does very little in relation to VPN connections. The only thing the ISA Server does when you run the ISA Server VPN Wizards is create the packet filters to support the VPN protocols. The Routing And Remote Access Service (RRAS) handles all other components of the VPN.

Some common VPN configuration and management errors you may encounter include:

  • Name resolution issues
  • IP addressing problems
  • VPN client configuration problems
  • VPN gateway issues
  • Authentication and encryption errors

Once you get a handle on these areas, you’ll be in good shape to have a smoothly running ISA Server VPN server.

Name resolution
Name resolution issues are common with ISA Server running a VPN server. The biggest problem occurs when the ISA Server administrator doesn’t have a network infrastructure to support the ISA Server solution. You can easily handle the problem by installing and configuring the appropriate network services.

If you have problems with NetBIOS name resolution, the obvious solution is to configure a WINS server. A WINS server is required if you want your VPN clients to resolve the NetBIOS names of machines on the internal network, because the VPN client cannot use NetBIOS broadcasts to resolve NetBIOS names on the internal network.

You can manually or automatically configure the VPN client with an address of a VPN server on the internal network. An alternative is to configure an LMHOSTS file on each VPN client. LMHOSTS files are an inferior solution because they aren’t dynamically updated. If machines on the internal network use DHCP for IP address assignment, the LMHOSTS file will be worthless. However, if you use static addresses for network servers, the LMHOSTS file can be a viable alternative.

An alternative to NetBIOS name resolution is DNS host name resolution. VPN clients can use a DNS server that is manually or dynamically assigned to them. They can query the internal network DNS server to resolve host names on the internal network. The problem most VPN clients run into when it comes to hosting name resolution on the internal network occurs when the VPN clients try to resolve unqualified names on the internal network. This becomes an issue when the VPN client wants to use DNS host name resolution to resolve the NetBIOS names on the internal network.

When the VPN client sends an unqualified request to the DNS server on the internal network, the DNS resolver on the client typically appends the VPN client’s domain name to the request. If the client is configured with an inappropriate domain name or no domain name at all, the request for name resolution will fail. If the internal network DNS server is configured to use a WINS server for name resolution, the request may succeed. But in that case, you should just assign a WINS server address to the VPN client in the first place.

When using ISA Server, VPN clients get their name server addresses from the settings on the internal interface of the VPN server or from a DHCP server on the internal network. If you configure a static address pooling for the VPN clients, the clients obtain their name server address from an internal interface on the VPN server. If the VPN server had multiple internal network adapters, you’d need to choose the one that has the name server addresses that you want assigned to the VPN clients. You can configure which adapter to use for name server assignment by looking at the Properties page of the VPN server. Click the IP tab on the Properties page. Make sure the Adapter field at the bottom of the IP tab is set to Internal as shown in Figure A.

Figure A
Select the interface that you want to assign name server addresses.

Your other option is to use a DHCP server on the internal network. When your ISA Server is configured to run as a VPN, make sure you install and configure the DHCP Relay Agent on the VPN server computer. VPN clients never directly communicate with a DHCP server because the VPN server doesn’t pass broadcast messages from VPN clients to the internal network. The DHCP Relay Agent will proxy for the VPN clients and allow them to receive DHCP options.

One thing that definitely won’t work without a WINS server is the browser service. If your users need access to a network server browser list, you must install a WINS server and configure the VPN clients to obtain the WINS server address.

IP addressing
VPN clients can get an IP address from a static address pool or from a DHCP server on the internal network. You can configure a static address pool in the same dialog box seen in Figure A. Make sure that the addresses in the static address pool aren’t already in use on the network and that they aren’t assigned to a scope on any of your DHCP servers.

You can also use a DHCP server to assign addresses to VPN clients. The RRAS server will obtain addresses from a DHCP server when the RRAS server starts up. The server will obtain more addresses when needed, however, the RRAS server doesn’t retain any DHCP options. The only way to assign DHCP options, such as WINS address, DNS address, and domain name, to VPN clients is to install and configure the DHCP Relay Agent on the VPN server.

You can assign VPN clients on-subnet and off-subnet addresses. On-subnet addresses are those that match the same network ID as the internal interface of the ISA Server. On-subnet addresses are easiest to manage, because VPN clients have a valid IP address for the network ID that the VPN server is directly attached to.

You can also use off-subnet network addresses for the VPN clients. In this instance, the VPN clients are assigned IP addresses that are not on the same network ID as the internal interface of the ISA Server. This can be a useful security measure. If the network routing infrastructure isn’t set up to support the off-subnet addresses that you assign the VPN clients, the only resources the VPN clients will be able to access are on the VPN server itself.

Whether you use on-subnet or off-subnet addressing on the VPN clients, it’s critical that you configure the routing table on the VPN server with the appropriate network IDs on your internal network. If there are networks you don’t want the ISA Server to reach, leave them out of the routing table, but all other networks must be included in the routing table. If you use off-subnet addresses for VPN clients, you must configure the routers on the network to forward responses to the off-subnet network ID to the internal interface of the VPN server.

In general, it is a good idea to configure the VPN clients to use off-subnet addresses. Doing so will prevent rogue VPN clients from accessing the internal network.

VPN client configuration
Windows VPN client software is configured slight differently, depending on the operating system. However, there is one setting common to almost all versions—the Use Default Gateway On Remote Network option. This is the default option. When selected, all packets for nonlocal networks are forwarded to the ISA Server’s VPN interface. This prevents VPN clients from accessing the Internet and the corporate network at the same time.

You can still allow VPN clients to access to the Internet through the ISA Server by configuring the VPN client’s browser to use the Web Proxy for the VPN connection. At the client, launch Internet Explorer and select Internet Options from the Tools menu. When the Internet Options screen appears, click the Connections tab. On the Connections tab, find an entry for the VPN connection (see Figure B). Select that connection and click the Settings button.

Figure B
Select the VPN interface on which you want to use the Web Proxy service

You can configure a proxy server address and port number in the connection’s Settings dialog box. Put in the IP address of the outgoing Web requests listener and port 8080, as shown in Figure C. You’ll also need to include the credentials required by the Web Proxy service if you’re requiring authentication for outbound access.

Figure C
Enter the IP address, port, and credentials when you use the Web Proxy service.

Either intentionally or out of curiosity, users may harpoon network security by disabling the Use Default Gateway On Remote Network option. When users disable this option, a network route is added to the VPN client’s routing table, but it isn’t a default route. The route sends requests for the classful network ID the VPN client was assigned for its VPN interface.

Let’s look at an example of a dial-up VPN client. The VPN client is assigned the IP address, and a default route for network ID is configured on the VPN client. All packets for that network ID (and all subnets of that network ID) are sent to the VPN server. All other nonlocal packets are sent to the ISP’s remote router. The VPN client then has a direct link to both the Internet and the corporate network and can become a gateway between the Internet and the corporate network.

A good way to prevent users from torpedoing internal network security is to design the IP addressing and routing scheme so that if users are able to set their VPN clients to not Use The Default Gateway On The Remote Network, they still won’t be able to access anything other than resources on the VPN server itself.

The best way to do this is to assign the VPN clients off-subnet IP addresses. For example, the internal interface of the VPN server is connected to network ID and the VPN clients are assigned IP addresses in the range. With this setup, those VPN clients configured not to use the VPN server as their default gateway will be able to access resources on the VPN server, but they won’t be able to access resources anywhere else on the internal network.

This is because when the client is configured not to use the default gateway on the remote network, the actual default gateway on the client points to the ISP (the Internet) or whatever default gateway the client already has set up. Any nonlocal requests—including those for network ID—will be forwarded to the existing default gateway, which obviously won’t work in getting to subnets on the VPN network. Even though the VPN server contains the proper routing table entries to forward requests to the network IDs on the internal network, the off-subnet VPN client won’t be able to take advantage of them because they are not using the VPN server as their default gateway.

VPN gateways
ISA Server includes a couple of nice wizards that allow you to create a local and remote VPN gateway. Gateways can join a remote office to a local corporate network. The Local Wizard is run on the machine receiving the call from the remote office VPN server. The Remote Wizard is run at the VPN server at the branch office.

These wizards work fine, except for a small problem with allowing both sides to initiate a connection. Only the remote office should have the capability to call the central office. If you configure both sides with the ability to dial one another, you’ll end up with a potential race condition when the VPN connection is dropped. Each server will try to dial the other simultaneously, preventing either from accepting an incoming connection.

You can prevent this problem by allowing only the central office to dial up the connection. After configuring the VPN gateways, go to the local VPN server and configure the machine to never redial a connection. The gateway-to-gateway VPN router configuration should have a passive side that receives calls and an active side that makes calls. On the passive side, remove the dial-up credentials from the demand-dial interface configured by the VPN Wizard.

Remember that your VPN gateway solution creates a routed connection to the remote network. You should treat the connection between the networks like you would any other routed connection. Configure your routing infrastructure to send packets for the appropriate network IDs to the network on the other side of the VPN gateway. This will prevent one of the most common communication failures between networks joined by the VPN gateway.

Design your network services infrastructure to support the routed networks. Place WINS, DNS, DHCP, and directory services with this routed architecture in mind. I often see questions from ISA Server administrators who wonder how to deal with NetBIOS and host name resolution for hosts on the other side of the network. You handle this problem as you would with any other routed network solution. Since the link between VPN gateways is not always reliable, you should install and configure redundant services on each side of the link and configure them to replicate with one another, using mechanisms appropriate for each network service.

Authentication and encryption
Your ISA Server configured to run VPN supports both PPTP and L2TP/IPSec VPN connections. ISA Server does not support pure IPSec VPN tunnels. This can create a problem when you want to configure a pure IPSec tunnel between the ISA Server computer and a third-party hardware VPN device. The only solution is to configure the third-party device to use L2TP/IPSec. If you do decide to use L2TP/IPSec, make sure that you configure compatible IPSec policies. Windows 2000 creates a default L2TP/IPSec policy.

You can disable the default IPSec policy and use an alternate policy. This will help if you want to use a preshared key between the VPN server and the black box, or you can configure a new IPSec policy to be used for your L2TP connections.

There is one major ISA Server configuration issue to take into account when implementing an L2TP/IPSec VPN: The Internet Key Exchange (IKE) required for the IPSec connection requires that you allow fragmented packets through the ISA Server. If you configure the ISA Server to block fragmented packets, all L2TP/IPSec VPN connection attempts will fail. This creates some risk because there are known exploits that take advantage packet fragmentation. Because of this, you might want to configure PPTP VPN connections instead. PPTP security is highly dependent on password complexity. If you’re able to implement a secure password infrastructure, then PPTP VPNs can be as secure as L2TP/IPSec VPNs.

Solve those VPN trouble spots
ISA Server is built with VPN connectivity in mind. In the majority of instances, ISA Server configuration is not the problem. The primary issue is either with the VPN configuration in RRAS or with problems with the underlying network infrastructure. When you know the main areas that can cause ISA Server’s VPN to fail, you can fix the problems as they appear.