Like any security mechanism, Internet Protocol Security or IPSec is imperfect. There are ways that hackers can defeat its protections in certain situations—for example, by using replay attacks (in which network communications are recorded and then replayed to “fool” the receiving computer), man-in-the-middle attacks, source routing exploits (in transport mode), and IP session hijacking (when you use IPSec without an authenticating header).

To help you overcome these threats, I will show you how to configure your Windows XP client computers to use IPSec when communicating with IPSec-enabled servers. You will learn how to set up basic IPSec connections, how to select the appropriate IPSec policy to be applied, and how to create and assign custom policies if none of the predefined policies included in XP meets your needs.

Microsoft default IPSec implementation
Microsoft has taken steps to make its implementation of IPSec as secure as possible, including use of short-term session keys that expire quickly. The XP IPSec implementation prevents man-in-the-middle attacks by authenticating identities after the Diffie-Hellman key exchange. A feature called Perfect Forward Secrecy (PFS) makes it impossible for a key used in protecting IPSec communications to be used to generate more keys. In addition, following recommended best practices can ameliorate many of IPSec’s vulnerabilities.


Microsoft allows you to set key lifetimes; that is, you can force new keys to be generated after a specific number of seconds. This makes the communication more secure because several different keys are used over the course of the transmission. Even if a hacker manages to crack one key, he’ll only have part of the message.

Configuring the Windows XP client to use IPSec
To configure a Windows XP client computer to use IPSec at the local level, you must be a member of the Administrators group. First, ensure that the IPSec Services service is enabled on the computer. In the Computer Management console (right-click My Computer and select Manage), expand the Services and Applications node and click Services. In the right pane, the status of IPSec services should be Started, as shown in Figure A.

Figure A
To use IPSec, the IPSec Services must be started.

If the service is not started, double-click its name and click the Start button on the General tab, shown in Figure B.

Figure B
You can start the IPSec services via the General tab on its properties sheet.

Next, you must assign an IPSec policy. To assign a policy locally, first create a new MMC by typing mmc in the Run box (Start | Run), and adding the IPSecurity Policy Management snap-in (select Add/Remove Snap-in from the File menu). In the Select Computer or Domain dialog box, ensure that the Local computer option is selected and click the Finish button. This will create the IPSec console shown in Figure C.


You can also manage IPSec policies on a remote computer, by creating an IPSec MMC and selecting Another computer, and then browsing for the name of the computer whose policies you want to manage.

Figure C
You can assign an IPSec policy via the IPSecurity Policy Management MMC.

By default, there is no IPSec policy assigned. To assign one of the three predefined policies (listed in the right console pane when you click IP Security Policies on Local Computer in the left pane), right click the policy you want to assign (for example, Client) and select Assign from the context menu. Under the Policy Assigned field, a “Yes” will appear.

Editing or creating IPSec policies
Usually, one of the predefined policies will meet your needs, but you can edit one of the policies to customize it if you like. To do so, double click the policy you want to edit. You can edit the key exchange settings by clicking the Advanced button on the General tab. This will display the Key Exchange Settings dialog box shown in Figure D.

Figure D
You can edit the key exchange settings of any of the predefined policies.

Here you can select to use PFS for the master key, change the interval at which new keys are authenticated and generated (in minutes or after a specified number of sessions) and select the security methods (DES or 3DES encryption algorithm, SHA1 or MD5 hashing algorithm for integrity, and Diffie-Hellman group 1 or 2).

Using the Rules tab, you can add or edit IPSec rules. The Create IP Security Rule Wizard makes this easy. On the first page of the Wizard, you’ll be asked whether to specify a tunnel endpoint (and the IP address of the endpoint if you elect to use tunneling), as shown in Figure E.

Figure E
The Security Rule Wizard begins by asking if you want to specify a tunnel endpoint.

The next page of the Wizard lets you choose the network type(s) to which the rule must be applied: LAN, remote access, or (the default) all network connections. Next, you can choose the initial authentication method. The default is Active Directory (Kerberos v5). However, Kerberos can only be used if the computer is a member of a domain. If it’s not, you’ll need to select another method. Alternatively, you can choose to use a certificate (you’ll have to specify the issuing certification authority) or a preshared key (in which case you must enter the character string that makes up the key).

The next page of the Wizard prompts you to select an IP filter list for the type of IP traffic to which the rule will apply. You can select to apply the rule to all ICMP traffic, all IP traffic, or add a custom list (this selection brings up another Wizard within the Wizard: the IP Filter Wizard).

Next, you select a filter action. The default actions are:

  • Permit unsecured IP packets
  • Request security (optional)
  • Require security

This completes the Wizard, and when you click Finish, your new rule will appear in the IP Security rules list on the Rules tab of the policy’s properties sheet, as shown in Figure F. You can check or uncheck it to specify whether it is to be used.

Figure F
When you complete the Security Rule Wizard, your new rule appears on the Rules tab.

Editing an existing policy will usually suffice, but if you want to create an entirely new policy from scratch, you can do so by selecting Create IP Security Policy from the Action menu of the IPSec Policy Management console. As you might have guessed, this invokes the IP Security Policy Wizard.

This Wizard starts by asking for a name and description for your new policy, then asks you to specify how the policy will respond to requests for secure communications from other computers. If you select to use the default response rule, you’ll be asked to set an initial authentication method for it (Kerberos, certificate, or preshared key). This completes the Wizard and it appears in the list of policies in the right pane of the console, along with the three predefined policies. It can now be edited or assigned.

The IPSec console also provides ways for you to manage the IP filter lists and actions, restore the default policies if you’ve changed them, and import and export policies (save them to a file).

Recommendations for IPSec best practices
IPSec is simple in concept, but complex in implementation. Microsoft recommends that before applying IPSec policies, you develop an IPSec plan (as part of your overall security plan). The first step is to evaluate how your sensitive information routes through the network and which computers have access to it. Specifically:

  • Determine the level of IPSec security you need. That is, decide whether you need to secure all traffic between all computers, or only the traffic to/from specific computers (or maybe only traffic to/from specific ports or using specific protocols). Decide whether to secure LAN traffic only, remote access traffic only, or both.
  • Determine the type of IPSec security you need. This means deciding whether you need authentication, integrity, confidentiality, or a combination of these.
  • Determine the level at which IPSec policies will be applied and managed: site, domain, OU, or locally.
  • Determine the encryption strength needed—whether DES will do, or whether you need the stronger 3DES. Decide what hashing algorithms to use.

Test policies before deployment
Now you can create an IPSec deployment plan and create policies based on your evaluation. Be sure to test the policies before putting them to work in a production environment. Be aware that incorrect (or overzealous) application of policies may result in an inability for computers to communicate. You should use a packet sniffer (protocol analysis software) to ascertain whether the data in transit is being successfully encrypted.