Networking

Understanding and troubleshooting virtual private networking

While VPN lets users connect to their LAN over the Internet, it can also require a lot of troubleshooting. In this Daily Drill Down, Debra Littlejohn Shinder tells you all you need to know to get your client to play nice with VPN.

It’s easy to understand why virtual private networking (VPN) is steadily increasing in popularity—VPNs are flexible and secure. In this Daily Drill Down, I’ll explain how VPNs work and how to troubleshoot common client-side configuration and connection problems.

Three ways to connect from the road
Over the past few years, it has become more and more important for workers to be able to connect to their company networks even when at home, on the road, or on location at clients’ sites. This can be accomplished in three basic ways:
  • A direct dedicated connection
  • A dial-up remote access connection
  • A VPN connection

The last is an attractive alternative for several reasons, including flexibility, cost-effectiveness, security, and ease of implementation.

All modern Microsoft operating systems—Windows 95 (with the Dial-up Networking 1.3 upgrade), 98, Me, NT 4.0, and 2000—include built-in support for virtual private networking, as will the yet-to-be-released Windows XP (code name “Whistler”). Tunneling protocols and VPN client functionality can be installed from the Windows installation CD, making a computer running any of these operating systems function as a VPN client. Although Microsoft has made VPN configuration as simple as possible, even providing wizards in the latest versions of Windows, establishing and using a VPN connection can still be a little tricky.

VPN overview
Before you can effectively diagnose and correct VPN problems, you need a basic understanding of how a VPN works. A VPN connection is made by “tunneling” through a public network (typically the Internet) to reach a private network. There are three basic components involved:
  • Encapsulation
  • Encryption
  • Authentication

Encapsulation
The link between the client and the private network emulates a point-to-point connection by encapsulating, or wrapping, the data packet (which can use any network transport protocol supported on the private network, including NetBEUI, IPX/SPX, or TCP/IP) inside a header that contains information for routing across the public TCP/IP network.

Encapsulation is performed by a tunneling protocol. There are two tunneling protocols supported by Microsoft for VPN connections: the Point-to-Point Tunneling Protocol (PPTP) and Layer 2 Tunneling Protocol (L2TP), depending on the operating system (see Table 1).

Tunneling protocols supported by Microsoft operating systems
Operating system Supports PPTP Supports L2TP
Windows 95 with Dial-up Networking 1.3 upgrade   Yes   No
Windows 98 Yes No
Windows NT 4.0 Yes No
Windows 2000 Yes Yes
Windows Me Yes No
Table 1

Encryption
Encapsulation makes it possible for the packets to travel across the public network. Encryption provides confidentiality for the contents, creating the element of privacy. It is possible to send unencrypted data through a tunnel, but this would be virtual networking, not virtual private networking. Encryption is performed by:
  • Microsoft Point-to-Point Encryption (MPPE) protocol—with a PPTP tunnel
  • IPSecurity (IPSec) protocol—with an L2TP tunnel

VPN data encryption is between the VPN client and VPN server. The availability of encryption depends on the user-level authentication method used for the connection.

Authentication
Authentication protocols supported by all Microsoft VPN clients include:
  • Challenge Handshake Authentication Protocol (CHAP)
  • Microsoft Challenge Handshake Authentication Protocol (MS-CHAP) version 1
  • Password Authentication Protocol (PAP)
  • Shiva Password Authentication Protocol (SPAP)

Additionally, Windows 2000, Windows 95 with the Dial-up Networking 1.3 upgrade, Windows 98 with Service Pack 1 or later, and Windows NT with Service Pack 4 or later support MS-CHAP version 2. Only Windows 2000 supports the Extensible Authentication Protocol (EAP).

Troubleshooting common VPN problems
A VPN connection requires that both the client and the VPN server be connected to the Internet. The client connection can be either via a direct dedicated connection or a dial-up connection to the client user’s ISP. The dial-up connection presents the first point of failure, so it is important to ensure that the client is properly connected to the ISP. Do this by checking the status of the dial-up connection. Attempt to ping an Internet host. If you are able to do so, you have a connection to the ISP. If not, you will need to troubleshoot your modem configuration and/or Dial-up Networking.

The most common VPN connection problems fall generally into one of the following categories:
  • Problems related to the Internet connectivity on one or both sides
  • Problems related to VPN server configuration
  • Problems related to VPN client configuration

This Daily Drill Down focuses on the last category, but if you are unable to detect a reason for the problem on the client side, the problem may lie in one of the other categories. Most VPN users will be running the Windows 9x/Me, NT 4.0 Workstation, or Windows 2000 Professional operating system on the client computer. Basic considerations are the same, although there are some differences in configuration dialog boxes. First, I’ll look at configuration of Windows 2000 clients and then point out differences (where they exist) in other Microsoft operating systems. As with any troubleshooting situation, you should first consider the most basic (and easiest to correct) possibilities.

Invalid credentials (logon failure)
You must have a valid username and password that will allow you to connect to the VPN server. Otherwise, you will receive this message:

Your credentials have failed remote network authentication

You will be prompted to re-enter your username and password (and logon domain, if applicable). Ensure that you are entering the proper account name and password.

Your user account must be set on the server to allow remote access (although this is a server configuration problem, it is one to be aware of if your credentials are rejected by the VPN server). If you’re sure the credentials you entered are correct, contact the network administrator or check the settings on the Dial-in tab of the user account properties sheet on the server.

Tunneling protocol configuration problems
If the VPN server does not support the tunneling protocol with which the client is attempting to connect, the connection will fail. Windows 9x/Me/NT clients support only PPTP as the tunneling protocol. However, when you configure a Windows 2000 client, you have three options: PPTP, L2TP, or Automatic. The last is the default (in which case the client will try to establish an L2TP connection first, then try PPTP).

If your VPN client is configured to use a different tunneling protocol from that supported by the VPN server, you will see an Error 678 message (No answer), as shown in Figure A.

Figure A
You may get an Error 678 message if the VPN server does not answer.


This error occurs because there are no ports on the server configured to answer for the specified tunneling protocol. You may also get this message if all the PPTP or L2TP ports on the VPN server are already in use.

When you use the Network And Dial-Up Connections Wizard to create a Win2K VPN connection, Automatic will be set as the server type (the wizard does not offer you a chance to select). However, you should check the protocol configuration in the Properties box for the VPN connection, as shown in Figure B, and change it to Automatic or to the protocol setting for which the server is configured to correct this problem.

Figure B
In Windows 2000, you can select a specific tunneling protocol.


Note that in Windows NT 4.0, only PPTP is supported, and you must install PPTP as a new protocol in the Network Properties dialog box, as shown in Figure C. The VPN connection is configured by adding a VPN adapter device from the Port box in the RAS Setup dialog box. You then create a dial-up networking entry to connect to the PPTP server.

Figure C
PPTP must be installed as a network protocol in Windows NT 4.0.


In Windows 98, you add Virtual Private Networking as a Communications component from Add/Remove Programs in the Control Panel. Making a VPN connection requires using two connections: a dial-up connection to the Internet and a PPTP connection to the private network. You configure the VPN connection with the Make New Connection Wizard by selecting Microsoft VPN Adapter as the connection device instead of a modem.

If the VPN server supports only L2TP connections, you will not be able to establish a VPN with a Windows 9x/Me or Windows NT client.

You may also see the same No Answer error message presented above if PPTP filtering is enabled in the server’s RAS configuration. Another possibility, if you are able to ping the server by name and address but you cannot establish a VPN connection, is that the server is filtering GRE packets. Again, this is a server configuration problem that cannot be corrected from the client side.

Authentication method configuration problems
In addition to a common tunneling protocol, both client and server must support a common authentication method. If the client is not configured to use an authentication method that the VPN server supports, you will see an Error 919 message, shown in Figure D.

Figure D
If there is no common authentication protocol, you will get an Error 919 message.


In Windows 2000, check the authentication protocols allowed by the client in Advanced Security Settings, accessed via the Security tab of the VPN connection properties sheet, shown in Figure E.

Figure E
The client must use an authentication protocol that is allowed by the server.


Note that the Windows 2000 client can be configured to use EAP or to allow any or all of the standard authentication methods supported by other Microsoft operating systems. If you have chosen to use EAP, you can choose either Smart Card Or Other Certificate-Based Authentication or MD-5 Challenge.

Remote access policies on the VPN server can also be responsible for this error message.

Encryption type mismatch
Another reason your VPN connection may fail is a mismatch between data encryption requirements on the client and server. In the Windows 2000 client’s Advanced Security Settings, you can choose one of the following:
  • No Encryption Allowed (The server will immediately disconnect during the negotiation if its own settings require encryption.)
  • Optional Encryption (A connection will be made regardless of the server’s encryption settings.)
  • Require Encryption (If the server is set not to allow encryption, it will disconnect.)

A mismatch will result in an Error 742 message, shown in Figure F.

Figure F
An encryption type mismatch will result in an Error 742 message.


As with authentication protocols, Windows 2000 encryption settings are changed via the Advanced Security Settings property sheet, as shown in Figure G.

Figure G
You can select No Encryption, Optional Encryption, or Require Encryption.


Because the default setting is to Require Encryption, if you get this error message, you should always check out the possibility that the VPN server is set not to allow encryption.

“Unreachable Destination” problems
In all of the problems discussed above, the client is able to contact the VPN server but is unable to negotiate a VPN connection because of configuration settings that do not meet the server’s criteria. Sometimes, you’ll see the Error 769 message (The specified destination is not reachable), shown in Figure H.

Figure H
If the destination VPN server cannot be contacted, you will get an Error 769 message.


There can be three causes of this error:
  • The VPN server may be offline. You should attempt to ping the server to ascertain whether it is online.
  • You may have entered the wrong server name or IP address for the VPN server in the client configuration properties. You should recheck the entry and ensure you have the proper name or IP address entered.
  • There may be a name resolution problem if you are trying to connect by server name. You should try connecting by IP address instead of server name.

If you can ping the server by IP address but not by name, this means that either the server does not have a registered domain name, or the DNS server on the network is down or not functioning correctly.
Note that if the VPN server has a dial-up connection to the Internet, it is likely that the ISP will assign it a different IP address each time an Internet connection is established. You will not be able to connect by IP address unless you know the server’s currently assigned address. More commonly, a VPN server will have a permanent (static) address and a dedicated full-time Internet connection.
Problems after connecting
If it appears that the VPN has connected successfully, but you are unable to access resources on the server or LAN, you can check the status of the connection in several ways.

A Windows 9x/Me/2000 client will have an icon in the system tray for each remote access connection. Right-click the icon and click Status. You will see a status box similar to that in Figure I. In the Activity section, note changes in the number of bytes sent and received to confirm that traffic is going across the VPN connection.

Figure I
You can determine whether traffic is moving across the VPN connection by checking its Status sheet.


The same properties box can be accessed via Start | Settings | Network And Dial-Up Connections. Again, right-click the VPN connection and select Status. In addition, the status for the VPN connection should show as Connected in the Network And Dial-Up Connections window, as shown in Figure J.

Figure J
Connection status is shown in the Status column of the Network And Dial-Up Connections window.


At the command prompt, run netstat. This will show a list of active connections, as shown in Figure K. In the Foreign Address column, you can recognize VPN connections by the appearance of the type of tunneling protocol (PPTP or L2TP) following the foreign address.

Figure K
The netstat command will show you all active connections, including VPN connections.


Note the two active VPN connections in Figure K—one to a VPN server named Mail.starblazer.tzo.com and one to Exeter.tacteam.net.

If your connection is established and you can access the VPN server, but you cannot browse the LAN or access resources on other computers, there can be several reasons for the problem:
  • The VPN server must be configured to allow access to the entire LAN, not just to the VPN server computer. This is a server configuration that cannot be corrected from the client side.
  • If you receive an Error 53 message, (Network path was not found) this may be because the client cannot resolve NetBIOS names. Ensure that the client has a WINS server assigned. This can be done manually in the client’s TCP/IP properties or via DHCP.
  • Your account may not have the proper permissions to access the resources on the server.

If you are unable to browse, attempt to connect to the network shares on computers inside the LAN using the UNC path (\\servername\sharename).

If you are using Windows 9x clients to log on to a domain, ensure that the workgroup name in the Network Identification properties is the same as the domain name.

Undesired disconnections
If you are able to establish the VPN connection, access resources, and browse the network, but the link is prematurely disconnected, check the following possibilities:
  • You may have the VPN connection configured on the client to hang up after a specified number of minutes of idle time. As shown in Figure L, you can set this value from one minute to 24 hours, or to never hang up (the default). Check this setting first if you are getting disconnected at a regular interval of time.

  • Figure L
    Disconnections can be caused by an idle time hang-up setting in VPN properties.

  • Your ISP may have idle time limitation rules or connect time limitation rules that automatically disconnect you after a specified period of idle time or a specified period of time online, even if active.

  • Firewall and proxy problems
    The firewalls and proxy servers that often protect our networks from unauthorized access can complicate VPN connectivity. Although you may be unable to correct these problems from the client side, you should be aware that if your client configuration appears to be correct and you are unable to establish a VPN connection, it may be due to firewall or proxy restrictions, or packet filtering rules on the router.

    For example, you may be unable to browse the LAN if NetBIOS packets are being filtered by the firewall or router. In this case, the network administrator needs to open UDP ports 137 and 138 and TCP port 139. You cannot reach a VPN server behind a firewall at all unless port 1723 is open and protocol 47 (GRE) is allowed.

    If you have an active Winsock Proxy client, you will not be able to create a VPN, because the Winsock client immediately redirects data to the proxy server before it can be processed as necessary by the VPN. You will have to disable the Winsock Proxy client. Microsoft’s Proxy Server does not support outbound PPTP requests from internal clients (behind the proxy). Gateway-to-gateway VPNs, in which the proxy server is the VPN client, can be established. Also, because IPSec does not work with address translation (which is the way Proxy Server provides Internet access to its clients via a single public IP address), you will not be able to establish an L2TP/IPSec VPN from behind the proxy server.

    In Microsoft’s new ISA Server, the Winsock Proxy client is called the Firewall client. ISA clients running the Firewall client software cannot establish a VPN because outbound PPTP is not supported. However, ISA clients that are configured as Secure NAT (SNAT) clients can be VPN clients.
    For more information about Microsoft’s ISA Server, see ISAserver.org.
    Conclusion
    Virtual private networking provides a cost-effective way to take advantage of today’s almost universal Internet connectivity to create a private “tunnel” to a company LAN or other network from a remote location. VPN connection failures and other problems are often easy to correct by modifying the client configuration. In this Daily Drill Down, I showed you techniques for troubleshooting and correcting common client-side VPN connectivity problems.
    The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

    About

    Debra Littlejohn Shinder, MCSE, MVP is a technology consultant, trainer, and writer who has authored a number of books on computer operating systems, networking, and security. Deb is a tech editor, developmental editor, and contributor to over 20 add...

    0 comments

    Editor's Picks