Those who are familiar with a PPTP VPN in Windows 2000 will find that an L2TP/IPSec VPN is quite similar but contains some more complicated settings and management. Along with configuring computer certificates, which I discussed in my last article, an L2TP/IPSec connection involves some in-depth work with the VPN settings and other configuration options. This article will introduce you to the more advanced approaches that will enable you to customize the security of your Win2K L2TP/IPSec connections. This will include:

  • ·        How the default L2TP/IPSec policies work.
  • ·        How to monitor the IPSec connections.
  • ·        How to override the default IPSec settings.

How the default L2TP/IPSec policies work
When you’re using Microsoft’s IP Security (IPSec) outside a VPN environment, you must assign a preconfigured IPSec policy to the computers. The Security Policy console (under Administrative Tools) allows you to view and edit these IPSec policies. However, by default, Microsoft uses a hidden, automatic IPSec policy for L2TP connections, which you won’t see in the Security Policy console. It is called the L2TP Rule, and you can see it only when it’s in use.

The default L2TP Rule policy is in use on the server when the RRAS server is listening on L2TP ports and on the remote workstation when the client tries to connect over L2TP/IPSec. If you stop the IPSec policy agent on the VPN server (for example, by typing net stop policyagent) after RRAS has initialized, you will delete this default policy. To re-create it, restart the policyagent service and then the RRAS service or reboot. The default L2TP Rule is automatically deleted on the Windows 2000 client whenever the L2TP/IPSec connection is terminated.

By default, a Windows 2000 client VPN connection will try an L2TP/IPSec connection first. If this fails, it then falls back to trying PPTP. This is why there is no need to change anything on the client’s connection properties if the defaults are still in use when you try to make an L2TP connection from the client. However, you might want to change this for security reasons so that only an L2TP/IPSec connection will be tried. If so, you will need to go into the connection’s Network properties and change the Type Of VPN Server I Am Calling setting from Automatic to Layer-2 Tunneling Protocol (L2TP).

You can check to see that an L2TP connection is being used on the VPN server by looking at the Ports folder in the RRAS console on the VPN server. Look for an Active status on an L2TP WAN Miniport, as shown in Figure A.

Figure A
RRAS showing an active L2TP connection

The RRAS console will tell you that an L2TP connection is being used, but it won’t tell you anything about the IPSec side of the connection. To see exactly what IPSec settings are being used, you’ll have to delve a little deeper.

How to monitor the IPSec connections
You use some of Win2K’s standard IPSec monitoring utilities to see what IPSec settings are being used for your L2TP/IPSec connections. This article assumes that you have a basic understanding of how IPSec connections work, along with their basic components. However, if you need some background information, these two resources are a good place to start:

You can see the L2TP policy in use with the IP Security Monitor. When you have a successful L2TP/IPSec connection, type ipsecmon from a command prompt on the RRAS server, and you’ll see the L2TP Rule policy. It should look similar to Figure B.

Figure B
The default L2TP/IPSec policy in use

This monitor gives you some (but not all) of the information on the current IPSec connection. To see all the information, you’ll have to use the Netdiag Windows 2000 Support tool by typing netdiag /test:ipsec /v at the command line. You’ll also have this level of information recorded in your Security Event log if you have enabled auditing for successful logons.

The policy filters on the VPN server are sensible ones that you probably shouldn’t change. You’ll find them under the Current Phase 2 SAs section when you use the Netdiag command. They are the source address(es) of the VPN server’s Internet NIC to any destination address and any source port from the VPN server to destination port UDP 1701.

However, what is interesting is that (as with any IPSec connection) the remote access client and VPN server can negotiate security options that will be used for the connection. The default L2TP Rule allows the VPN server to offer 16 security preferences. (The equivalent options can be found under the Security Methods tab when using the Security Policy console.) To see all offers, type netdiag /test:ipsec /debug on the server.

The first match between client and server will be used, so if your Windows 2000 client and Windows 2000 VPN server offer the same level of encryption (e.g., both support only 56-bit encryption), the resulting security methods used will be data encryption (ESP) with DES and Cipher Block Chaining (CBC), together with MD5 as the chosen algorithm method. This matches the ESP DES/CBC HMAC MD5 in Figure B. If both server and client support strong encryption (i.e., they both have Win2K SP2 installed), the resulting policy will be ESP 3DES/CBC HMAC MD5.

If the encryption levels are not the same on the server and the client, the lower one will be used. So if you want the highest encryption level on your L2TP/IPSec connections, ensure that both the server and all clients support 128-bit encryption. The easiest way to do this is to install SP2 or to install the High Encryption Pack if you are running a pre-SP2 machine with 56-bit encryption. However, you should realize that connections using 3DES are slower and demand more processing on the server.

You may be surprised when looking through the full list of 16 “offers” in Netdiag that there are more secure security methods on the list that will not be used by default because they are farther down the offer list. For example, you can use both Authenticated Headers (AHs) and ESP to ensure that the header information (addresses) is not changed in transit, and you can use SHA1, which is a stronger algorithm than MD5. However, both of these come with the overheads of additional processing, and if you use AH as well as ESP, you will also need to open Protocol ID 51 on your firewall.

The least secure offer on the list has AHs without encrypting the data at all. This is not most people’s idea of a virtual private network, but there may be times when this option is necessary for political reasons—for example, when the data is being transferred in a country where encryption is banned. However, if you specifically want to ensure that all connecting remote clients will encrypt their data, having this offer automatically listed (albeit at the bottom of the offer list) may worry you because you cannot change this default offer list. Fortunately, you can customize your IPSec settings to prevent the possibility that this offer will be used.

How to override the default IPSec settings
You may be wondering how it is possible to use any of the other offers if a Windows 2000 remote client to Windows 2000 VPN server uses the same policy, which always results in matching ESP with 3DES and MD5. Because the first match between client and server will be used, a VPN client that doesn’t use the Microsoft default L2TP Rule may be configured with different security options, so you can’t predict which of the 16 offers will be used. Because of this uncertainty, or a desire to use IPSec settings that are different from the default, you may have good cause to change the IPSec options on the VPN server.

For example, you may have deployed SP2 on all of your Windows 2000 computers for the security patches but do not want the extra processing of 3DES, and you want to use DES instead. Or you may want to use the strongest combination possible, AH and ESP, using the SHA1 algorithm. Or you may decide you don’t want the risk of potentially offering a VPN connection that doesn’t encrypt data.

If you decide to go this route, you’ll need to disable the default L2TP/IPSec policy and create one manually that matches the security options you want to use. To disable the default policy, add a new registry key (REG_DWORD) of ProhibitIpSec and set the value to 1 under this Windows Registry key.

Next, reboot the computer. You can do this just on your VPN server to ensure that only the security settings you want will be used and then let the client work through its default offer list until a match is found. Or you can do the same on the client side so that both sides use only one offer.

Now, you need to configure your own IPSec policy and assign it. Make sure that you change the default authentication from Kerberos to certificates. Use the filters previously mentioned, select the security methods you want, and use Netdiag to ensure that your options are being implemented. Make sure you also choose to rekey every so often and select your own settings for this or use the sensible defaults in the L2TP/IPSec policy, which are every 3,600 seconds or every 250,000 bytes.

This article has provided information that should help you understand, monitor, and tailor Microsoft’s L2TP/IPSec connections for a more secure VPN connection. You’ll find additional information on Microsoft’s VPN site.

Have a comment or a question?

We look forward to getting your input and hearing about your experiences regarding this topic. Post a comment or a question about this article.