SolutionBase: Discover the new IPSec features in Windows Server 2003

Learn what's new with IPSec in Windows Server 2003.

When Microsoft released Windows 2000, it included a new protocol called IPSec. IPSec was designed to encrypt network traffic and to guarantee that a packet could not be altered in route or replayed at a later time. Although the original version of IPSec was pretty good, it did have room for improvement. With the release of Windows Server 2003, Microsoft included many enhancements to the IPSec protocol while still allowing it to be backward compatible with the original version.

A little warning up front
After reading this article, you might be anxious to modify your IPSec policy to include all of the new IPSec features. Keep in mind, though, that many of the features I'll be discussing are not compatible with Windows 2000 or with Windows XP. Therefore, the only way that you'll be able to use some of the new IPSec enhancements is for communications between Windows Server 2003 machines.

Diffie-Hellman enhancements
One of the challenges of shared key communications is that both computers involved in a communication must be able to access the same shared key. The problem is that you can't send that key across the network or security will be compromised. There have been a lot of different algorithms designed to solve this problem, with the most popular being RSA (Rivest-Shamir-Adleman). The ironic thing is that the even older Diffie-Hellman key-generation algorithm is more secure than RSA.

The Windows 2000 and Windows XP implementations of IPSec supported Diffie-Hellman groups 1 and 2. The group numbers refer to the number of bits of keying material, which ultimately affect the strength of the key itself. Groups 1 and 2 are 768 and 1024 bits, respectively. However, with Windows 2003, Microsoft has included a 2048-bit Diffie-Hellman algorithm.

The 2048-bit algorithm is exponentially more secure than the 1024-bit algorithm, but it does have some negatives. First, a 2048-bit algorithm takes longer to compute than a 1024-bit algorithm. The other problem is that it is only compatible with Windows Server 2003. The strongest algorithm that can be used in communications between Windows 2000, XP, and 2003 is Diffie-Hellman group 2 (1024-bit).

One of the neat things about Diffie-Hellman is that it does not perform any type of authentication. At first, this might sound like a bad thing, but think about it. Instead, Diffie-Hellman provides the keying material for all other encryption keys. When two machines want to initiate communications with each other, the machines publicly exchange keying information.

Windows XP and Windows 2003 also add a hash for even greater security. Once the keying information has been exchanged, both machines have all the information they need to be able to generate keys. Only when the computers are able to generate their own encryption keys does the authentication process begin. This way, authentication credentials are never transmitted in an unencrypted format.

NETSH integration
A feature that isn't really a part of IPSec itself, but is new to Windows Server 2003, is the ability to configure IPSec from a command prompt via the NETSH utility. In case you aren't familiar with NETSH, it has been around since Windows 2000 and allows you to configure many of Windows' features directly from a command prompt. The NETSH interface is commonly used as a scripting interface for those who want to automate specific tasks.

Although NETSH has been traditionally used mostly by scripters, there are several IPSec features that can be configured only through NETSH, not through the GUI. A couple of examples are computer startup security and persistent policy security. I'll talk about these features in more detail later. In the meantime, if you want to control these and other features through NETSH, simply open a command prompt on a Windows Server 2003 machine and enter the NETSH command followed by the IPSec command. This will load the NETSH utility and put it in IPSec mode. From there, just enter a question mark to display a list of the available commands.

Computer startup security
One of the new IPSec features that can be controlled only through NETSH is computer startup security. To get a feel for how this works, think about what happens when you boot up a Windows 2000 Server that has IPSec enabled. IPSec is enabled through a computer-level group policy. During the boot process, a lot of different things happen before the group policy is applied. In fact, applying the security policy is pretty much the last step before the system allows an administrator to log in. In some cases, it's even possible for someone to log in before the policy finishes loading.

Although IPSec does a good job of protecting network traffic, IPSec can't offer any protection until it has been initiated. Your system is completely vulnerable from boot-up until the IPSec policy is initiated.

In Windows Server 2003, though, Microsoft allows you to use NETSH to perform stateful packet inspection during system startup. When this feature is enabled, the only types of traffic that are allowed are DHCP traffic, outbound traffic from the server, and responses to requests issued by the server. If you prefer, you can block all network traffic until IPSec is active. There is also an exceptions option that allows you to specify types of traffic that you want to allow during boot-up. This is useful if you want to allow DNS traffic, for example.

Persistent policies
A similar feature that can also be applied only through NETSH is persistent policies. Normally, an IPSec policy is applied either through the local security policy or through an Active Directory-level group policy. Suppose, however, that the applicable policy became corrupted. In this case, the policy would not be applied, thus leaving your computer vulnerable. You might never even know that the policy hadn't been applied unless you were watching your log files or the IP Security Monitor.

A persistent policy takes care of this problem. The idea behind a persistent policy is that it can secure your system regardless of whether a local or domain-based IPSec policy is applied. The persistent policy is loaded at boot-up and remains in effect unless overridden by another policy.

Default traffic exemptions
Although IPSec is designed to encrypt all network traffic flowing into and out of a computer, there is some traffic that is not and cannot be encrypted. In the Windows 2000 and Windows XP implementations of IPSec, these traffic types include Internet Key Exchange (IKE), Kerberos, RSVP, and all broadcast and multicast traffic.

In Windows 2003, Kerberos and RSVP traffic are no longer exempt from IPSec filtering. IKE traffic is exempt from IPSec filtering because IPSec filtering can't occur until after IKE has done its job. IPSec doesn't attempt to negotiate security associations for multicast and broadcast traffic, but you can configure multicast and broadcast filters that block or permit these types of traffic.

Certificate mapping
Another new feature that is supported only by Windows Server 2003 is certificate mapping. The basic idea behind this is that if you have an enterprise certificate authority, you can assign a certificate to each machine on your network. You can then configure IPSec to either allow or deny specific clients access to various network hosts. You can implement these restrictions based on domain membership or on which certificate authority issued the computer's certificate. You can also grant or deny access to a specific computer or group of computers.

Setting up certificate-based access control is fairly easy to do. Once certificate mapping is enabled, IKE links each computer account in the Active Directory to its corresponding certificate. This results in IKE receiving an access token, which is basically just a list of the rights that have been granted to the computer. Now that IKE has an access token, you can set the Access This Computer From A Network or Deny Access To This Computer From The Network permissions.

As I mentioned, you can only use certificate-based mapping to control access to servers running Windows Server 2003. Furthermore, because of the way that certificate mapping matches computer accounts in the Active Directory with certificates, you can't use certificate mapping to grant or deny access to a computer that exists in a different forest.

An interesting side note about IPSec's use of certificates is that you can now exclude the name of the certificate authority from certificate requests when certificate authentication is used to establish IPSec communications between two computers. The advantage of leaving out the name of the certificate authority is that in doing so, you prevent information such as the computer's domain membership from being disclosed.

NAT Traversal
Arguably the most significant new feature in IPSec is NAT Traversal. In case you aren't familiar with NAT, it stands for Network Address Translation. The idea behind NAT is that since IP addresses are in short supply, an entire company can share a single IP address. Computers inside the company are assigned bogus addresses.

Whenever a computer on the company's private network needs to access an external resource, such as the Internet, the request is sent to the NAT firewall. The NAT firewall, which holds the company's one legitimate IP address, then forwards the request on behalf of the computer that actually made it. When a remote computer responds to the request, it responds to the NAT firewall, since that's where the request appears to external hosts to have come from. The NAT firewall then passes the response to the computer on the private network for which it was intended.

With Windows 2000 and Windows XP, it was impossible to send IPSec-encrypted traffic across a NAT. However, Windows Server 2003 supports NAT Traversal of IPSec traffic, so long as the NAT supports UDP (User Datagram Packets). The IKE protocol can automatically detect the presence of a NAT firewall. If a NAT firewall is detected, IPSec-encrypted packets are UDP-ESP encapsulated so that they can flow across the NAT.

You might be wondering why this feature is significant, since you obviously aren't going to use IPSec to surf the Internet. The idea is that even if a Windows server 2003 is behind a NAT firewall, it can still establish VPN communications or access a remote RAS server while maintaining IPSec encryption. Granted, you would probably never perform such an operation directly from the server console, but now that the technology exists, you can expect to see IPSec NAT Traversal capabilities built into Microsoft's next desktop operating system.

Monitoring IPSec
IPSec is initiated through group policies, and the problem with group policies is that they have traditionally been difficult to configure. Setting up an individual group policy is no big deal, but group policies are hierarchical in nature. Group policies can be applied at the local computer, domain, site, and Organizational Unit levels.

Since there are so many different levels where group policies can be applied, conflicts can and often do occur between group policies. Windows has a set of rules that it uses to resolve these conflicts, but the problem is that it isn't always easy for us to determine why Windows assigned a specific policy.

To help administrators figure out where various policy elements were coming from, Microsoft introduced a new tool in Windows Server 2003 called Resultant Set of Policy (RSoP). Since IPSec is initiated through a group policy, there is an IPSec extension to RSoP. After running RSoP, you can view detailed information about the IPSec policy that is being applied. This information includes such things as filter types, filter rules, and authentication methods.

The IPSec extension to RSoP isn't the only tool that you have for monitoring IPSec. When Microsoft released Windows 2000, it included a little-known tool called IPSECMON.EXE. This simple utility let you find out whether or not IPSec was indeed encrypting network traffic. Over the years, I have grown quite fond of IPSECMON.EXE, and I was really surprised to see that it didn't exist in Windows Server 2003.

However, Microsoft has replaced IPSECMON.EXE with a new tool called the IP Security Monitor onsole, shown in Figure A. In creating this tool, Microsoft has basically rewritten IPSECMON.EXE to make it work within a console. Microsoft then added support for all of the new IPSec features that I have talked about.

Figure A

As you look at the IP Security Monitor console, one of the first things you might notice is that just below the IP Security Monitor container is a server listed by name. The reason for this is that the IP Security Monitor Console can monitor the IPSec statistics for multiple computers rather than just for the local computer, as was the case with IPSECMON.EXE.

Another nice new addition to the console is the ability to view individual IPSec policies. You can use the console to view things such as the policy name, description, modification date, store, path, and OU.

Yet another new feature is the ability to use DNS name resolution for filter and security association output. At first, this might not seem like a big deal, since IPSECMON.EXE did some host name resolution. The difference, though, is that IPSECMON.EXE only resolved the names of hosts on the local network. The new IP Security Monitor Console allows you to resolve host names from across the Internet. This is important because Windows Server 2003 supports using IPSec over a NAT firewall.

One last new feature
One last new feature that I want to talk about is the filter search. As you probably know, IPSec policies are based on filters and rules. For example, a filter rule might dictate that traffic flowing from and addressed to a specific address must be encrypted. Although filters and rules tend to work well, the very nature of Active Directory allows conflicts to occur.

When conflicts occur between filters, one filter will take priority over another, and the lower priority filter will be ignored. This can cause IPSec to behave unexpectedly. If you notice IPSec behaving strangely, you can use the new console to search for specific filters or rules. This allows you to locate the filter that is causing the unexpected behavior.