Virtual private networks (VPNs) are growing in popularity, and the reason is obvious: The ability to connect to a private LAN by “tunneling” through the public Internet provides both convenience and cost benefits. This is especially true if the connections are of long duration or over very long distances. This is called an access VPN, and can work across analog or ISDN dial-up lines, DSL, or cable modems. Intranet and extranet VPNs can be used to connect corporate offices at different locations to one another using a dedicated connection, or to link to the sites of partners or customers.

A great deal has been written about VPN tunneling and popular tunneling protocols such as Point-to-Point Tunneling Protocol (PPTP), Layer 2 Forwarding (L2F) and Layer 2 Tunneling Protocol (L2TP). IPSec can also perform tunneling, but it is more commonly used to provide security for an L2TP tunnel. These tunneling protocols make virtual networking possible.

Using a tunneling protocol, a client computer anywhere in the world can connect to a server running the same protocol and access the entire company LAN on which the server resides (provided both are configured appropriately). The key is that both the client and the server are connected to the global Internet. The network is virtual because there is no direct connection from the client or server.

What about the second part of the equation? A virtual network is useful and may save money (because a client can dial up a local ISP instead of making a long-distance call directly to their company’s RAS server), but it’s not secure. Data packets are traveling across a vast public network and could be intercepted and read or changed en route. Before virtual networking can be used for sensitive or mission-critical communications, you must find a way to make it private. That’s what VPN security protocols are all about.

In this Drill Down, I will show you how VPN security works, discuss the protocols that keep the data private as it travels through the tunnel, and address the question of “Just how safe it is to send your data through a VPN?”
There are many ways to construct a VPN tunnel, including both software- and hardware-based implementations. Some use open source software, such as Secure Shell (SSH), and others—such as CheckPoint’s FWZ Encapsulation—are proprietary. Each VPN vendor can offer its own solution, as there are no universal standards.
Components of a secure connection
The tunneling process is called encapsulation because the tunneling protocol, also called the encapsulating protocol, creates the tunnel through which the passenger protocol (for example, PPP in a dial-up connection) travels. A carrier protocol carries the encapsulated packet. IP is typically the carrier protocol, as it routes packets across the Internet.

The tunnel is the conduit by which the data travels over the public network, but it does not secure the data itself. Security of data communications involves several issues, including:

  • Verifying the origin of the data—making sure that the apparent sender really sent it. This is called authentication.
  • Ensuring the integrity of the data—making sure that it has not been modified during its travel between sender and recipient. This is also called packet authentication.
  • Providing confidentiality for the data—making sure that if an unauthorized party does intercept it, it cannot be read.

A secure VPN connection provides all three of these services using security protocols.

Protocols protect privacy
Just as a VPN requires a tunneling protocol to establish the virtual network, it also requires separate protocols to provide the privacy. These include:

  • Authentication protocols
  • Encryption protocols

Authentication protocols are used to validate the identity of a user or computer and ensure data integrity. Encryption protocols provide confidentiality.

The authentication and encryption protocols used for a VPN connection depend on the particular implementation by the vendor of the VPN solution and settings selected by the VPN user (on the client side) and the network administrator (on the server side).

For a VPN client and VPN server to establish a connection and communicate over an internetwork, they must support at least one common authentication protocol and one common encryption protocol.

VPN authentication
There are different authentication levels associated with VPNs. Two of them are user-level authentication (PPP authentication) and machine-level authentication, which uses the Internet Security Association and Key Management Protocol (ISAKMP).

User-level authentication
When a VPN client attempts to make a connection to a VPN server, the server will use a PPP user-level authentication method to confirm the client’s identity based on the user’s credentials (account name and password or smart card and PIN). In addition, the VPN server must verify that the client has the proper permissions to establish the connection (this is called authorization or access control).

With mutual authentication, the process goes both ways; the client also authenticates the server to verify its identity and protect against server masquerading.

Some user-level authentication protocols include:

  • Password Authentication Protocol (PAP), which uses plain text passwords to authenticate the client’s identity.
  • Shiva PAP (SPAP), which is used to authenticate Shiva clients.
  • Challenge Handshake Authentication Protocol (CHAP), where the server “challenges” the remote client to supply authentication credentials. Message Digest 5 (MD5) is used to hash the message, allowing the hash value to be sent across the network instead of the actual password.
  • Microsoft CHAP (MS-CHAP), which is a version of CHAP developed by Microsoft to authenticate Windows clients. MS-CHAP v2 is a mutual authentication protocol by which both the client and the server prove their identities to one another.

The first two are supported by Cisco’s L2F implementation; the third is associated with Microsoft’s PPTP.
Hashing is a nonreversible method of applying an algorithm (formula) to a string of characters, such as a password, to disguise the original data. It is called nonreversible because you cannot reverse the formula and recover the original data.
Authentication can also be provided by Remote Authentication Dial-In User Service (RADIUS), which is an industry-standard protocol that requires clients to send user and connection information to a RADIUS server, which authenticates the client and authorizes the connection request.

PAP is the least secure authentication method because if an unauthorized person uses a packet sniffer (for example, the Network Monitor software built into Windows NT/2000) to capture the data as it travels across the network, he or she will be able to view the contents of the packets and read the password. PAP is not recommended for VPN authentication.

Machine-level authentication
When the IPSec encryption protocol is used, the authentication of the client and server machines is done using machine certificates. The ISAKMP protocol is used to create a security association, and the Oakley key-generation protocol is used to generate and manage the authenticated keys that are used to secure the data.
Digital certificates use cryptography to verify identity, while making it difficult for an unauthorized person to intercept, alter, or spoof (forge) data. Certificates use public and private keys (a key pair) to sign messages; the sender’s public key is included in the certificate. The sender signs the message using his or her private key. Using the public key that matches it, the recipient can verify the sender’s identity.
VPN encryption
The data that passes through a VPN tunnel is encrypted to provide confidentiality. When it reaches its destination, it is then decrypted so it can be read. If someone captures the data packets in transit, he or she will not be able to read the messages without access to the encryption key. The key is used to lock the data, and the longer the key is, the more secure it is. Just as a ten-letter password is more difficult to guess than one with only three letters, a 128-bit key is more difficult to crack than a 40-bit key.

The protocol used to secure VPN data depends on the encapsulation protocol that is used to build the tunnel. For example, Microsoft PPTP VPNs use the Microsoft Point-to-Point Encryption (MPPE) protocol. The specifications for Cisco’s L2F (see RFC 2341) do not specify an encryption protocol.

Both Microsoft and Cisco now support L2TP as the preferred tunneling protocol. Both implementations operate in conjunction with the IP Security Protocol (IPSec) to provide encryption.
L2TP is an Internet standard, defined in RFC 2661. IPSec is described in RFC 1825.
IPSec was developed to add security to data that travels across a network (not only VPN connections) and is capable of providing both authentication and encryption. The components of IPSec are:

  • Authentication Header (AH)
  • Encapsulating Security Payload
  • Security associations (SAs)

Let’s take a look at how these components work.

IPSec Authentication Header (AH)
AH can provide authentication and integrity between a set of hosts or between a set of gateways. Both ends of the connection must implement AH. It is important to understand that AH does not provide confidentiality of data; it can still be read if intercepted. However, it does provide protection from modification. The packet is signed, and the signature ensures that the identity of the sender is known and that the data has not changed since it left the sender.

The authentication header is placed between the IP header and the TCP/UDP header on the data packet. The entire packet is signed (see Figure A).

Figure A
AH signs the entire packet, providing authentication and integrity but not confidentiality.

IPSec Encapsulating Security Payload
Encapsulating Security Payload (ESP) can provide authentication, integrity, and confidentiality of data traveling between two or more hosts or two or more gateways that have implemented ESP. VPNs often utilize gateway-to-gateway encryption, which protects the data while it is traveling on the public Internet. In this case, data is not encrypted while on the private network (see Figure B).

Figure B
When ESP is used between two gateways, data is encrypted only when traveling over the Internet.

ESP and AH can operate in two modes: transport mode or tunnel mode. In tunnel mode, ESP creates a tunnel to provide privacy for tunneled packets. The packets can be encrypted using Data Encryption Standard (DES) or 3DES (also called Triple DES), although encryption is not required if only authentication and integrity are desired and confidentiality is not required.

ESP in transport mode is used to provide the security for a tunnel created by L2TP. When used in transport mode, ESP does not sign the entire packet. It protects the data, but not the IP header (see Figure C).

Figure C
In transport mode, ESP does not sign the entire packet.

IPSec security associations
IPSec creates a security association (SA) to define the security services and keys that will be used to secure a communication between two hosts or gateways. The security association is like a contract between the sending and receiving computers (source and destination) that lays out the terms or rules for the transaction.

The Internet Engineering Task Force (IETF) has established a standardized way for this process to take place, using two technologies:

  • Oakley

ISAKMP manages the security associations and negotiates the security policies. Oakley generates the authenticated keys that are used to protect the data. ISAKMP/Oakley is known as the Internet Key Exchange (IKE). Cisco Systems prepared IETF drafts specifying standards for IKE and made a version of IKE available at no charge via the Internet.

The security association is established by a two-part process:

  • Key exchange
  • Data protection

During the key exchange step, the communicating computers create the ISAKMP SA. Oakley protects the identities during this step. Policy is negotiated, and the computers exchange information that allows the generation of a shared secret key, which is generated by the Diffie-Hellman protocol. The computers must then authenticate the key information exchange (note that the keys themselves are not exchanged; only the information that is used to generate the shared “master” key).

The second step, data protection, begins with the negotiation of a pair of SAs that are called the IPSec SAs (to differentiate them from the ISAKMP SA). Policies are negotiated for the new SAs. There are two because one is used for inbound communication and the other for outbound. The ISAKMP SA protects the negotiation. Multiple IPSec SAs can be protected by one ISAKMP SA.

The IPSec authentication and encryption process
Once the SAs have been established, the sending computer will use the outbound IPSec SA to sign the packets (to provide integrity) and encrypt the data (for confidentiality). The packets will then be transmitted through the tunnel to the destination computer.

The destination computer will use the inbound SA and corresponding key to verify the integrity signature and decrypt the data, and then the decrypted data will be passed by TCP/IP to the appropriate application.
If there is a firewall, proxy server, or router functioning as a security gateway between the communicating computers, packet filtering must be configured to allow the IPSec packets to go through.
Security is a major concern for VPN implementations because the data passes across the public Internet to reach the private network. There are many ways to provide protection for tunneled data. One of the most common security protocols in use today is IPSec. IPSec can provide authentication, integrity, and confidentiality. It can even be used in tunnel mode as the encapsulation protocol that creates the virtual tunnel, however, it is more often used in transport mode to provide encryption of data that passes through an L2TP tunnel. Microsoft and Cisco, along with other vendors, support L2TP and IPSec as the most secure and effective means of implementing VPNs.

In this Daily Drill Down, I discussed how VPNs work, and went under the hood to examine the security protocols that put the “private” into virtual private networking.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.