Setting up and managing an L2TP/IPSec VPN in Windows 2000 is quite different in many respects from working with a standard PPTP VPN. So it’s not surprising that troubleshooting these connections also requires some unique tactics, as this article will demonstrate.

Microsoft’s L2TP/IPSec connections usually fail for two main reasons:

  • Problems with certificates
  • Internet device problems (e.g., routers, switches, firewalls, or NAT)

Other potential problems include:

  • Straining server resources
  • Interoperability with other systems

Problems with certificates
My article “Configuring certificates for an L2TP/IPSec VPN” worked through an example of how to use your own in-house CA to issue computer certificates required for L2TP/IPSec connections in Windows 2000. If you suspect that certificates may be to blame for your L2TP/IPSec connections failing to connect, try the steps in this article. Alternatively, you can use Microsoft’s testing site to install a computer certificate from Microsoft’s online CA.

Here are a few other things to check:

  • Verify that the date/time is correct on the client and the VPN server (and the issuing CA, if using an in-house CA).
  • Open the Certificates console on the client and verify that the CA path is installed, if using an in-house CA. You can confirm that it exists under Trusted Root Certificates Authority | Certificates or by checking that the computer certificate is listed and valid under Personal | Certificates.

If you still suspect that certificates may be the problem, an option to confirm this is eliminating them and using password authentication instead of certificates. This is possible only if you disable the default L2TP/IPSec policy and configure your own IPSec settings. One of the advantages of using your own IPSec policy is that you can change the authentication method from certificates to passwords.

You may decide that this is a configuration you actually want to use all the time rather than just for troubleshooting because it allows you to use L2TP/IPSec and bypass all the overheads of installing, managing, and maintaining your own Certificate Authority. Perhaps you cannot justify the expense of using a third-party Certificate Authority, or you have a non-Microsoft L2TP/IPSec client that is compatible but can use only passwords.

However, Microsoft does not endorse using computer password authentication for L2TP/IPSec connections. It argues that it is not a secure implementation because passwords are always vulnerable to guessing and/or cracking and will be stored in the registry or Active Directory as part of the IPSec policy. So remember that it is possible to use Microsoft’s L2TP/IPSec connections with password authentication instead of certificates, but you’re unlikely to get a sympathetic hearing from Microsoft if you report problems with them.

To use passwords instead of certificates for your L2TP/IPSec connections, you’ll have to disable the L2TP policy on both server and clients and then configure and assign your own IPSec policy as described in my article “Customize the security of L2TP/IPSec connections.” But specify password authentication and type in the password you want to use. For production use, don’t forget all the rules about choosing secure passwords (at least eight characters, mixture of alphanumeric and nonalphanumeric, mixture of cases, etc.).

If you need help setting up this policy, there are step-by-step instructions in the Microsoft Knowledge Base article Q240262, “How to Configure a L2TP/IPSec Connection Using Pre-shared Key Authentication.” This article will also help if you’re configuring your custom L2TP/IPSec policy with certificates. For the Authentication Method, instead of selecting a preshared key, select Use A Certificate From This Certificate Authority (CA) and select the CA by browsing.

Internet device problems
Check any Internet device that might be blocking the connection or changing the packets. Typically, this will be a firewall or a NAT server but can also include a faulty switch that is occasionally corrupting packets or a router that isn’t forwarding Protocol ID 50.

In the first article in this series, we said that Microsoft’s L2TP/IPSec is not compatible with NAT. However, some L2TP implementations are NAT-friendly (e.g., Cisco’s version) because they use a different implementation. See Microsoft’s VPN FAQs for more information on the Microsoft implementation and how it differs from other vendors’. With the Microsoft implementation, it may be possible for NAT to allow one client to connect with L2TP/IPSec but not allow subsequent connections, so you should always connect at least two remote clients before celebrating.

Even if you’re not using NAT and think you have configured your firewall correctly (to allow UDP port 500 and Protocol ID 50), if your L2TP/IPSec connections are not working, it may be a good idea to eliminate the Internet side of the equation by trying to make a VPN connection from a client machine on your LAN. If your connections still don’t work, at least you have narrowed it down to something on the client or server rather than attempting to verify all the possible Internet issues (hardware devices, ISP services, bandwidth, firewall, etc.).

To test this connection, you’ll need to temporarily rearrange your network so that there’s a standard Ethernet connection between the VPN server’s Internet adapter and your testing workstation. Assign the workstation a static IP address in the same range as the VPN server so that routing is not required and then make sure that you can successfully ping between the client and server.

Next, create a new VPN connection on the Windows 2000 Professional machine that doesn’t automatically dial the ISP connection first. (This is not the default.) Attempt to connect your newly created VPN connection. PPTP and L2TP connections work just fine over Ethernet since all they care about is a valid underlying TCP/IP connection. So, if your client and server are configured correctly, you should have a good L2TP/IPSec connection. It’s important to verify that the connection is an L2TP connection. (On the server, check the Active Ports under the RRAS console, or on the client, check the connection’s Status Details). If you have a successful L2TP/IPSec connection when connecting this way but not when you connect a similar client over the Internet, it’s time to start inspecting your Internet devices.

Straining server resources
L2TP/IPSec connections consume more server processing power—specifically, for the data encryption—than PPTP connections. This will be especially true if you are using strong encryption, which will happen by default if both client and server can support it. Keep an eye on the VPN server’s CPU usage, either with the Performance utility running as a service or, more crudely, with Task Manager’s CPU Performance figures. If you discover the processor slowing down, you have several options, including adding another processor, using a network card that offloads some of the IPSec processing, or disabling the default policy and specifying DES encryption instead of 3DES.

Interoperability with other systems
If you are hoping to use either the client side or the server with a different vendor’s implementation of L2TP/IPSec, check for interoperability issues and determine whether they can be configured to communicate, and when. Check my previously mentioned article on customizing security for information on the default IPSec settings that will be tried, the order they’ll be tried, and how you can disable them and define your own policy if necessary.

However, you can’t change Microsoft’s implementation of L2TP/IPSec, which uses IPSec in Transport mode (not Tunnel mode), and the UDP port number of 1701 cannot be changed. If the third-party vendor’s implementation also uses Transport mode and port 1701 for the IPSec side of the connection, chances are you can configure the custom IPSec settings to match if the defaults do not work.

Additional guidance
L2TP/IPSec connections must establish an IPSec connection before the tunnel (L2TP), so if the IPSec connection fails, the tunnel is never even attempted. Enable logon auditing and check the Event Viewer’s Security log for IPSec errors such as negotiation timeouts (could be lack of a valid certificate or a packet being blocked by network devices). Make a note of the actual error logged and then look it up on Microsoft’s Knowledge Base. You may also find these TechNet articles useful:

I hope this article has provided some useful tips to help troubleshoot your Microsoft L2TP/IPSec connections and, combined with my previous articles, has given you a good basic understanding of how Microsoft’s implementation of L2TP/IPSec works.

Have a comment or a question about L2TP/IPSec VPNs?

We look forward to getting your input and hearing your experiences regarding this topic. Post a comment or a question about this article.