Encryption of data is an important facet of computer security in today’s business environment, and it’s useful to many small office users as well. In a previous article, “Make EFS part of your Windows XP security plan,” I discussed how data stored on disk can be protected using the Encrypting File System (EFS) that’s built into Windows 2000 and XP. However, EFS does not protect data that is sent across the network. If you attempt to send an EFS-encrypted file across the network, you’ll find that the data can be read using sniffer (protocol analysis) software. Thus, you’ll need a different way to protect data in transit.

Recognizing this security vulnerability, Microsoft included its implementation of Internet Protocol Security (IPSec) in Windows 2000 and then in Windows XP. IPSec is not a proprietary encryption mechanism like EFS; it is an industry standard for encrypting data sent across an IP network and is part of the IPv6 (the next generation of IP) specifications. The Internet Engineering Task Force (IETF) specifications for IPSec, which are aimed at providing interoperability across different platforms and operating systems, can be found in RFCs 1825 and 2401. To help you understand the implications of IPSec with Windows XP, I will provide you with a brief overview of what IPSec is and how it works in Windows.

NOTE

In this article, I’ll be discussing IPSec as implemented with IPv4, the current version of the Internet Protocol. Keep in mind that some of IPSec’s vulnerabilities have been improved upon in IPv6.

IPSec: What it is and how it works
IPSec is not actually a single protocol; rather, it is a set of protocols that operate at the network layer (Layer 3) of the OSI networking model. The layer at which it operates is important because this is what allows IPSec to work regardless of what application is being used to send the data across the network. IPSec is not the only means of encrypting data during transmission, but methods that operate at higher levels (such as SSL) require that applications using them be written to recognize the encryption protocols. With IPSec, the application is never aware that encryption is occurring at the lower levels. On the other hand, SSL connections can be made quickly (if you have the proper applications) and don’t require network-level configuration as IPSec does. The two technologies are generally used for different situations.

Note

IPSec can be used alone for secure network transmissions, or it can be used in conjunction with the Layer Two Tunneling Protocol (L2TP) for establishing virtual private network (VPN) connections. In the latter case, L2TP creates the tunnel (virtual connection) and IPSec provides the privacy by encrypting the data that will travel through the tunnel.

The components of IPSec
Let’s look at the parts and pieces that make up IP Security. These consist of two main protocols, along with several other protocols, algorithms, and components used by the main protocols. Different implementations of IPSec can use different algorithms, but all use the two main security protocols (sometimes called traffic security protocols) and a group of cryptographic key exchange protocols for generating and managing the encryption keys.

AH and ESP: The IP security protocols
The IPSec security protocols are the Authentication Header (AH) protocol and the Encapsulating Security Payload (ESP) protocol. Either can be used separately for a particular IPSec connection, or both can be used together. They serve different but related purposes. Security protocols generally provide one or more of the following:

  • Confidentiality of data (encryption of data so that it cannot be read by unauthorized persons)
  • Authentication of the origin of data (verification of the identity of the sender)
  • Integrity of the data (assurance that the data hasn’t been changed between the time it left the sender and the time it reached the destination)
  • Antireplay (also called replay prevention; prevents intercepting a transmitted message and replaying it to gain access at a later time)

The most important thing to remember about AH is that it does not encrypt data. It does provide authentication and integrity, and it provides this for the entire packet, including the IP header information. ESP provides confidentiality by encrypting the data itself. It can also provide authentication and integrity, but generally only for the data and not for the IP headers. The encryption algorithm used will determine how secure the data is. Microsoft’s implementation of IPSec uses DES or 3DES.

IPSec authentication can be based on digital certificates in a public key infrastructure (PKI) environment—the most secure method—or preshared keys (an agreed-upon string of characters) can be used. Preshared keys are less secure because the key is stored within the IPSec policy. Both security protocols can operate in either tunnel mode or transport mode. Tunnel mode generally provides gateway-to-gateway protection. Transport mode provides end-to-end protection.

Other IPSec components: Security associations, algorithms, and a driver
IPSec uses a type of connection called a security association, which is a stream of IP traffic with AH or ESP protection. If both protocols are used together, two separate SAs are created. An SA is a one-way (simplex) connection, so if IPSec is used to secure communications in both directions, this requires two SAs—one for each direction of transmission.

Each SA is identified by the combination of the traffic security protocol (AH or ESP), the destination IP address, and a security parameter index (SPI), which is a unique value created to differentiate between multiple SAs coming into a computer at the same time. The SA defines the authentication and encryption algorithms and the shared session key that will be used for a communication.

The Diffie-Hellman algorithm allows two parties to exchange keying information without having to send the actual key across the network. With the algorithm, both can generate identical keys individually. As mentioned earlier, ESP uses the Data Encryption Standard (DES) algorithm (and a stronger variant, 3DES) for data encryption. The IPSec driver is the software component that applies filtering actions (based on the IPSec policy) to permit or deny transmissions and matches SAs with the correct packets based on the SPI.

IPSec policies
Microsoft’s IPSec is policy-based. IPSec policies consist of rules that specify:

  • Types of IP traffic to be examined by IPSec (the filter list).
  • How IP traffic is to be treated, that is, whether specific types of traffic are to be permitted, blocked or secured (the filter action).
  • What authentication method(s) should be used.
  • Whether the traffic is to be tunneled (and the IP address of the endpoint of the tunnel if it is).
  • The type of connection (dialup and/or LAN).

Three predefined policies can be applied. The client policy allows the computer to use IPSec for secure communications when the computer with which it is communicating requests it. The server policy causes the computer to always request IPSec-secured communications, but if the computer on the other end is not IPSec capable it will accept nonsecured communications. The secure server policy causes the computer to require secured communications. If the computer on the other end is not IPSec capable, no communications will be accepted.

You can edit these policies or create custom policies using the IPSec MMC. Policies are assigned to Group Policy objects and can be applied to sites, domains, or OUs in an Active Directory network environment. In a nondomain environment, the local Group Policy object will be used to configure IPSec behavior.

Secondary IPSec protocols
In addition to the aforementioned protocols, IPSec uses other protocols that include the following:

  • Internet Security Association and Key Management Protocol (ISAKMP): This is an application layer protocol that specifies how SAs are established and negotiated. (This is a standard defined in RFC 2408.)
  • Internet Key Exchange (IKE): This is a standard method of centralizing the management of SAs and generating and managing shared secret keys.
  • Oakley key generation protocol: This defines how keying material is derived. ISAKMP is the framework within which IKE and Oakley are implemented.