With almost every computer networked, at least on a part-time basis, a special concern is how to secure data as it travels across that network. When an individual machine or local network is connected to the Internet, security becomes even more crucial. Disk encryption protects files from the prying eyes of another who logs on to the local computer, but what happens if you send a file across the network after encrypting it on the disk? A peek inside the packets with a sniffer (protocol analyzer), such as the Windows 2000 Server’s Network Monitor, will reveal a nasty surprise: The data is readable.

Exchanging sensitive information across a network, especially a public network, requires a security method that will protect the data in transit. That’s where Internet Protocol Security (IPSec) comes in. IPSec is a set of protocols that allows you to sign and encrypt data to be sent across an IP network, and authenticate and decrypt the protected packets on the receiving end. Windows 2000 Professional and Server include IPSec. In this Daily Drill Down, I will provide you with a basic understanding of how IPSec works in Windows 2000 and how to configure your Windows 2000 computer to use it.

What is IPSec?
IPSec is a set of protocols and cryptography-based services that work together to protect data from unauthorized access or tampering when it is sent across an IP network. IPSec provides three basic services:

  • Authentication: Confirmation of the origin of the IP packet; verification that the purported sender actually sent it
  • Integrity: “Signing” of the packet to ensure that the data has not been changed in any way between the time it left the sender and the time it was received at the authorized destination
  • Confidentiality: Encryption of the data to render it unreadable without the correct key

Windows 2000’s implementation of IPSec provides a high level of security, using a combination of algorithms and keys to encrypt the data so that it will be unreadable if intercepted along its route.

IPSec security protocols
IPSec uses two protocols to accomplish these tasks:

  • Authentication Header (AH): This signs the entire packet, providing authentication and ensuring integrity of the data. AH does not encrypt the data. It can be used alone when you need to confirm the identity of the sender and protect data from modification, but confidentiality is not required.
  • Encapsulating Security Payload (ESP): This provides authentication and integrity and also encrypts the data for confidentiality. ESP does not usually sign the entire packet (unless used in tunneling mode as described below), so only the data itself is protected; the IP header is not.

AH and ESP can be used together to provide the signing of the entire packet along with encryption of the data.

IPSec modes of operation
Both AH and ESP can operate in either of two modes: tunnel mode or transport mode. IPSec in tunnel mode is used to create a tunnel for a virtual private network (VPN). IPSec can be used in transport mode to encrypt the data inside a tunnel created by the Layer 2 Tunneling Protocol (L2TP). The tunneling process is technically termed encapsulation. For more information about IPSec in the context of VPNs, see my article entitled “Putting the ‘Private’ in Virtual Private Networking.”

In tunnel mode, IPSec only provides gateway-to-gateway or server-to-server protection. In transport mode, IPSec provides end-to-end security (from the originating computer to the final destination).

IPSec standards
IPSec is not a Microsoft invention; in fact, it is an Internet standard, defined by the Internet Engineering Task Force (IETF) in RFC 2401, “Security Architecture for the Internet Protocol.”

Advantages of IPSec
There are other methods for securing network-transmitted data, such as Secure Sockets Layer (SSL) or link layer encryption. However, SSL and many other network security methods operate at the higher layers of the OSI reference model, which requires that applications sending or receiving the secured communications must be designed to work with SSL. (These are called SSL-aware applications.)

A big advantage of IPSec is that it operates at the Network layer (also called Layer 3). This means applications do not have to be specially written to take advantage of IPSec. Protection of IP traffic, as well as upper-level TCP/IP suite protocols such as TCP, UDP, and ICMP, can be afforded transparently.

Link layer encryption works at the Data Link layer; its drawback is that it does not provide end-to-end protection on a routed network.

IPSec security associations
IPSec must be supported on both the sending and the destination computers in order to establish a secure exchange of data. These two systems first create a security association (SA), which is a negotiated agreement about how the data will be protected and exchanged.

A security association is made up of keys (generated by the Oakley service on each computer) and policies, which define the mechanisms for protecting the communication. A computer can have multiple simultaneous SAs. For example, a server being accessed by multiple clients might establish an SA with each.

The SA is created according to IETF standards using the Internet Security Association and Key Management Protocol (ISAKMP) and the Oakley key generation protocol. Together, these are known as the Internet Key Exchange (IKE).

Configuring IPSec on Windows 2000 Professional
The first step in configuring a computer running Windows 2000 Professional to use IPSec is to open the TCP/IP properties sheet for the network connection over which you want to use IPSec. (This must be a connection that uses the TCP/IP protocol.)

From the Start menu, select Settings | Network And Dial-up Connections and then right-click the connection you want to configure. Choose Properties and then on the General tab (under Components used by this connection), select Internet Protocol (TCP/IP) and click the Properties button (see Figure A).

Figure A
Configure the connection’s TCP/IP properties to use IPSec.

On the TCP/IP Properties page (see Figure B), click the Advanced button at the bottom of the page.

Figure B
Click the Advanced button to configure IPSec.

Now, click the Options tab (see Figure C) and select IP Security under Optional Settings. Then, click the Properties button.

Figure C
On this tab, select IP Security and click the Properties button.

To enable IPSec secured communications, select the Use This IP Security Policy radio button (see Figure D) and select an IPSec policy from the drop-down list.

Figure D
Select the Use This IP Security Policy radio button and choose a policy from the drop-down list.

Only members of the Administrators group can set IPSec policies. If the options are disabled and you cannot change them from the local computer, this is probably because the computer running Windows 2000 Professional is a member of an Active Directory domain and is receiving IPSec policies from Active Directory.
By default, you can select one of three predefined IPSec policies. These are:

  • Client (Respond Only)
  • Server (Request Security)
  • Secure Server (Require Security)

If you choose the Client policy, IPSec will not secure communications unless the destination server requests or requires it. This setting would be appropriate if the client is on an intranet, where most communications don’t need to be secured.

If you choose the Server policy, the computer will attempt to negotiate a secure communication when another computer initiates an exchange. However, if the computer on the other end is not able to do so (for example, if it’s a Windows NT 4.0 machine, which does not support IPSec), the computer will accept unsecured communications.

If you choose the Secure Server policy, the computer will accept and send only secured communications. If the computer on the other end is not IPSec-enabled, however, all traffic will be rejected. This setting should be used if the computer transmits data that is very sensitive or confidential.

Creating, modifying, and managing IPSec policies
Microsoft provides an IP Security Policy MMC snap-in for managing policies. Although the default policies will meet the needs of many organizations, you can modify them or create custom policies to fit your needs.

IPSec policies can be applied either locally or via Active Directory (in a Windows 2000 domain) using group policies. In this Daily Drill Down, I’ll focus on management of local policies.

You can create a custom MMC with the IPSec snap-in (see the sidebar on Creating an IPSec MMC), or you can access the policies via Start | Control Panel | Administrative Tools | Local Security Settings, as shown in Figure E.

Figure E
Local IPSec policies can be accessed via the Local Security Settings MMC.

The three predefined policies appear in the right details pane by default, as do any custom policies you create. Note that the figure shows the default policies and one custom policy.
To create a new MMC in Windows 2000 Professional, select Start | Run and type mmc in the run box. In the new, empty MMC console, open the Console menu at the top left and select Add/Remove Snap-In from the context menu.
On the Standalone tab, click the Add button. This opens the Add Standalone Snap-In dialog box. Scroll down through the list of available standalone snap-ins and select IP Security Policy Management. Click the Add button.
Because you are creating an MMC to manage local IPSec policies, select the Local Computer radio button and then click Finish. Click Close on the Add Standalone Snap-In box. Now click OK. The IP Security Policies node will now appear in the left pane of the snap-in.
You can save this console by selecting Save As on the Console menu. By default it will be saved in the Administrative Tools folder.
You can change the console mode by selecting Console | Options. Choose Author Mode if you wish for users of the MMC to have full access to all its functionality, including the ability to add or remove snap-ins and create new windows. Select User Mode—Full Access if you want users to be able to use all commands and have full access to the console tree but you want to prevent them from adding or removing snap-ins. Select User Mode—Limited Access (Single Window Or Multiple Window) if you want users to be able to access only the areas of the console tree that were visible when the console was saved.
IPSec policies are made up of filters and filter actions, and you can select the protocol(s) to which they will be applied.

Creating a new IPSec policy
To create a new IPSec policy, right-click IP Security Policies On Local Machine in the left console pane and choose Create IP Security Policy, as shown in Figure F.

Figure F
You can create your own custom IPSec policies using the IPSec MMC.

This will invoke the IP Security Policy wizard, which will walk you through the steps of creating a new, custom policy as follows:

  1. The wizard will first ask you to provide a name and description for the new policy.
  2. On the next screen, the wizard will ask you to specify how the policy should respond to requests for secure communications. By default, the default response rule will be activated, which is used when no other rule applies. The default response rule specifies that the computer must respond to requests for secure communication in order to establish IPSec-protected communications.
  3. The third screen will ask you to set an initial authentication method for the rule. By default, the Windows 2000 Kerberos v5 protocol is used. You can instead choose to use a certificate (in which case you must specify a certificate authority to issue the certificate), or you can select to use a preshared secret key, which is merely a string of characters that must be shared between the two communicating computers.
  4. Finally, you will be asked to click Finish to create your new policy.

After creating the new policy, you can edit its properties. (You can edit the properties of a policy at any time by double-clicking it in the right console pane or right-clicking and selecting Properties.)

Creating and editing security rules for the new policy
You can add and edit rules for this policy by using the options on the Rules tab, shown in Figure G.

Figure G
To add or edit IPSec rules for the policy you have created, click the appropriate button.

You can choose to use the Add wizard by selecting the check box in the lower-right corner. The wizard will walk you through the steps of creating a security policy that specifies how and when security will be used, based on criteria such as:

  • The source computer
  • The destination computer
  • The type of IP traffic

When the data packets for a communication match the criteria specified, one or more security actions will be performed. These actions are configured as you go through the steps of the Security Rule wizard, which are as follows:

  1. You must first specify whether this rule will cause an IPSec tunnel to be created. IPSec tunneling is used to create a virtual private network link, usually in situations where the other computer does not support L2TP tunneling. If you specify that a tunnel will be created, you must provide the IP address of the computer that will serve as the endpoint of the tunnel. (By default, a new rule does not specify a tunnel.)
  2. Next, you will be asked to select the type of network connection to which the rule is to be applied. You can choose from the following: All Network Connections, Local Area Network (LAN) Connections, or Remote Access Connections. (The default setting is all connections.)
  3. You will then be asked to specify an initial authentication method for the rule, selecting from the same three options (Windows 2000 Kerberos, certificate, or a preshared key) discussed above.
  4. The IP Filter List configuration sheet asks you to choose the type of IP traffic to which the rule will apply. Default options are:
    All ICMP Traffic
    All IP Traffic

    You can add additional filters by selecting the Add button on the IP Filter List screen. This will invoke the Filter wizard. You can configure these filters very specifically. You can specify that the rule apply to a specific IP address or subnet, a specific DNS name, your own IP address, or any IP address. You can also specify that the rule apply to any of the following protocol types: EGP, HMP, ICMP, RAW, RDP, RVD, TCP, UDP, XNS-IDP, or that it will apply to any protocol. If you select a protocol that uses a port (such as TCP or UDP), a port number will also be specified.

  5. The last step is to select a filter action for the rule. Default actions you can choose from include:
    Permit (This allows unsecured packets to pass through.)
    Request Security—Optional (This negotiates security; it will accept unsecured communications but always responds using IPSec. It will also allow unsecured communications if the other computer is not IPSec-aware.)
    Require Security (This will not allow unsecured communications with non-IPSec-aware computers.)

Although the wizard allows you to specify only one initial authentication method, a rule can support multiple authentication methods. To add additional methods, you must edit the rule after completing the wizard.
Creating a custom filter action
If none of the predefined filter actions fit your needs, you can create a custom filter action using—you guessed it—the Filter Action wizard. In creating the new filter action, you can specify whether you want the computer to be able to communicate with computers that do not support IPSec. On the IP Traffic Security sheet, you can also choose the IPSec protocol that will be used by this filter action (ESP or AH). You can even specify the integrity and encryption algorithms to be used by AH and ESP, and how often a new key will be generated, by choosing the Custom option.
Both AH and ESP can be used together; you can edit the properties of the filter action to add a second security protocol after you complete the wizard.
Assigning the new policy
Before your computer can use a policy to establish IPSec secured communications, you must assign the new policy. By default, no policies are assigned.

To assign a policy, right-click it in the right details pane of the MMC and select Assign, as shown in Figure H.

Figure H
You must assign the IPSec policy for the computer to use it.

Now the word Yes will appear in the column labeled Policy Assigned. To stop using a policy, right-click it and select Unassign.

IPSec is a useful feature included in the Windows 2000 Professional and Server operating systems that allows you to sign and encrypt data that you send across a network or the Internet. Using the IPSec security protocols, AH and ESP, you can provide authentication, integrity, and/or confidentiality of network communications in which your computer running Windows 2000 Professional participates. In this Daily Drill Down, I have discussed the uses of IPSec, its basic components, and how it works. I showed you how configuring your computer running Windows 2000 Professional to use IPSec is made easier by a variety of wizards that allow you great flexibility and control over IPSec policies.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.