The process of setting up connections for a virtual private network has been greatly enhanced and simplified by software solutions for Windows NT/2000, NetWare, and Linux/UNIX, as well as by hardware solutions offered by vendors such as Cisco and CheckPoint.

However, configuring VPN connections to pass through firewalls, proxy servers, and routers continues to bring many network administrators to their knees in exasperation and submission to the gods of the network cloud. Thus, we are going to review how to configure VPN servers to make connections in concert with your stoic network defenders.

VPN server geography
One of the first decisions a network engineer has to make when configuring a VPN server is where to place it in relation to the network’s firewall. As Figure A shows, there are essentially three options for placing a VPN server.

Figure A

The most common approach is to place the VPN server behind the firewall, either on the corporate LAN or as part of the network’s “demilitarized zone” (DMZ) of servers connected to the Internet.

Geography is extremely important when configuring and troubleshooting VPN connections that pass through firewalls. It lets you know which interfaces on the firewall will need filters assigned to them to allow VPN traffic. We’ll talk about filters at length in the next section. The thing to understand about geography and firewalls is that filtering occurs on the firewall’s external interface—the interface that connects to the Internet.

As I mentioned above, the most common place for a VPN Server is behind the firewall, often in a DMZ with mail servers, Web servers, database servers, and so on. The advantage of this placement is that it fits cleanly into the network’s current security infrastructure. Also, the administrator is already familiar with how to route traffic through the firewall and only has to become familiar with the ports needed by the VPN server. However, the other two options have benefits as well.

Placing a VPN server in front of the firewall can lead to greater security in some cases. Remember that a VPN allows users who are external to the network to feel like they are sitting on a machine inside the network. A hacker who hijacks a connection to a VPN server that is inside the firewall will be able to do some serious damage. However, if you have a dedicated VPN box that sits outside the firewall and that is only capable of sending VPN traffic through the firewall, you can limit the damage a hacker can do by hacking the VPN box. This option also allows you to limit the resources authenticated VPN users can access on the local network by filtering their traffic at the firewall. However, one vulnerability with this scenario is that the traffic between the firewall and the VPN server is not encrypted.

The third option is to colocate your VPN server on the same box as your firewall. In this case, the VPN server is still logically behind the firewall, but depending on its capability and utilization, it can complement a firewall very well, since both are essentially performing routing functions. This works nicely, since in most businesses, firewall/proxy services use more resources during the daytime hours, and VPN services use more resources during the evenings. However, keep in mind that having multiple services functioning on one box always involves management and troubleshooting challenges.

Understanding firewall and filter functionality
There are two types of filters and three types of firewalls to be aware of when configuring VPN connections. Filters come in two basic flavors:

  • Packet filtering
  • Application filtering

A firewall can engage in packet filtering, application filtering, or both. Filtering involves accepting or denying TCP/IP traffic based on source and destination address of packets, TCP/UPD port utilization and other TCP/IP headers information, and specific user and computer details in advanced firewalls.

Packet filtering
A packet filtering firewall merely examines traffic at the network layer (Layer 3 of the OSI reference model) and accepts or rejects it based mainly on source and destination addresses. Although a packet filtering firewall can do some blocking based on TCP and UDP port numbers, in most cases, it isn’t the best solution. However, packet filtering does provide speed, simplicity, and transparency.

Another important VPN troubleshooting tip deals with network address translation. If the Internet router or any router between the firewall and the VPN server is providing NAT, it will probably break the VPN tunnel and cause your connection to fail. The VPN server should have an Internet IP address on the external interface and not an internal IP address assigned by a DHCP server or hiding behind NAT. Most of the time you will get this Internet IP address from a subnet assigned to you by your ISP.

A packet filtering firewall is usually placed on a router and is managed through basic access control lists, which can be challenging to configure and manage. Here’s a common VPN problem to watch out for: Many administrators set up their VPN servers, configure their firewalls, and discover that they still can’t connect. They eventually realize that the ACL on their Internet router is filtering the VPN traffic and dropping the packets.

Application filtering
An application gateway firewall involves what is commonly known as proxy services and functions at the higher layers of the OSI reference model. This type of firewall offers more extensive, customizable features, such as user-level access control, time-of-day access control, and advanced auditing and logging.

It typically readdresses traffic so that it looks like it’s coming from the firewall rather than from the internal machine. In this manner, these firewalls act as a “proxy” on behalf of the internal network instead of providing a direct connection between internal and external networks, as you have with simple packet filtering firewalls. It also focuses on managing and controlling access to TCP/IP applications such as FTP, HTTP, rlogin, and so on.

Packet filtering and application filtering
Stateful inspection firewalls combine packet filtering and application filtering. They also employ a more secure firewall technique called dynamic packet filtering. With regular packet and application filtering, a port such as port 80 for HTTP is opened by the firewall and remains open for incoming and outgoing traffic. This presents a network vulnerability that hackers can exploit.

However, stateful inspection firewalls open and close ports as they are needed for traffic, drastically decreasing vulnerability to external attacks. Most popular firewalls, such as Microsoft Proxy Server 2.0, Network Ice’s ICEpac, and the leading UNIX solutions, use dynamic packet filtering.

Allowing VPN traffic
Now that you can see how various firewalls function, hopefully you can identify several places on your network where your VPN connection could be tripped up. Let’s see what filters you need to set up on these firewalls in order for VPN traffic to pass through them. In terms of protocols, we’ll cover VPN connections made using PPTP or L2TP over IPSec. We will begin with VPN filters at Layer 3 of the OSI reference model and work our way up to Layer 7.

When we look at receiving VPN traffic at Layer 3 we need to examine both the router that provides Internet access and the VPN server’s external interface. In some cases, the VPN server may have an external interface that connects directly to the Internet, such as an ISDN adapter. The router and/or the VPN external interface must be configured to accept TCP/IP connections from the VPN clients and/or VPN servers that will be connecting to it from the Internet. Thus, the access control lists (which manage filters at Layer 3) must be configured to allow incoming traffic from the IP addresses of these clients and servers. For remote VPN servers that are connecting, this will probably be a real IP, which will be easy to configure. However, for remote clients who are probably using a dial-up connection to an ISP and getting a different IP address each time, this is more challenging. If you have a restrictive IP access policy in place, you can get the range of IP addresses this client could use from his or her ISP or figure it out by deduction after a few connections. The other option is to allow access to all IP addresses by default and let upper-level filters accept or deny their packets based on application criteria.

When we get to Layer 7 (the application layer), we need to look at setting up filters to allow PPTP or L2TP with IPSec traffic based on the ports that they use. PPTP uses TCP port 1723, as well as IP protocol ID 47 for GRE (generic route encapsulation) tunnel maintenance. For the most part, if you are using a commercial firewall solution, you’ll only need to worry about setting up the PPTP filter for port 1723. But if you’re working with more complex firewall systems and do-it-yourself servers, such as Linux, you’ll need to be aware of the GRE port. Microsoft solutions such as Proxy Server 2.0 and the forthcoming Internet Security and Acceleration Server 2000 have predefined “PPTP receive” and “PPTP call” filters. These generally work pretty well.

Remember, you will need to be aware of the geography of your VPN server in relation to your firewall. For example, if your VPN server is behind your firewall, which connects to the Internet via a Cisco router, and you are receiving connections only from individual VPN clients (and not remote servers), you’ll set up a firewall filter to accept incoming traffic on port 1723 or simply select the predefined “PPTP receive” with a Microsoft solution. You’ll also need to go into the Cisco router and make sure that there are no access control lists filtering the VPN traffic.

As for L2TP with IPSec, the same principles apply, but it uses UDP port 1701 for L2TP and UDP port 500 for IPSec’s IKE (Internet key exchange). IPSec also uses IP Protocol port 50 for ESP (encapsulation security payload)—the equivalent of GRE for PPTP—but it doesn’t require a filter because the ESP header is typically removed by IPSec during routing before it hits the firewall.

Hopefully, the principles we reviewed here will enable you to better understand where your VPN connection could be running into snags in connecting through firewalls, proxy servers, and routers. We didn’t try to provide a step-by-step how-to on configuring firewalls and filters because of the vast configuration differences in the various hardware and software platforms, as well as the myriad different network typologies that are possible. However, you should be able to locate information on configuring filters and access control lists for your specific hardware and software platforms on the vendors’ Web sites. It also wouldn’t hurt to offer a sacrificial NIC or 100baseT cable to the networking gods before attempting your configuration.

Subscribe to the Cybersecurity Insider Newsletter

Strengthen your organization's IT security defenses by keeping abreast of the latest cybersecurity news, solutions, and best practices. Delivered every Monday, Tuesday and Thursday

Subscribe to the Cybersecurity Insider Newsletter

Strengthen your organization's IT security defenses by keeping abreast of the latest cybersecurity news, solutions, and best practices. Delivered every Monday, Tuesday and Thursday