Understanding the Win2K implementation of IPSec

IPSec is one of the exciting new technologies implemented as part of Windows 2000. See how IPSec works to ensure the identity, integrity, and confidentiality of network traffic and why you might want to use it to secure sensitive LAN communications.

One of the many exciting new features introduced in Windows 2000, and perhaps one of the better ones, is support for IPSec. You may have heard some of the talk about IPSec—probably in relation to VPN—but you may not know the full story about how IPSec can secure network communications in Windows 2000. Don't worry. I'm going to show you the ins and outs of IPSec in Win2K and tell you why you may want to implement it on your LAN. We’ll cover these topics:
  • What is IPSec?
  • How does IPSec work?
  • Do I need IPSec?

IPSec defined
IPSec (an acronym for Internet Protocol Security) is commonly misunderstood to be a protocol itself, which, technically, isn't true. The Internet Engineering Task Force (IETF) developed a framework of standards designed to secure communications over both public and private networks. These standards are collectively referred to as IPSec and have been published in a number of RFCs (2085, 2104, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2410, 2411, 2451—to look these up, go to the IETF Web site).

Historically, encryption has been an Application Layer (Layer 7 of the OSI reference model) or Presentation Layer (Layer 6) function, which allowed for data to be transferred over an insecure network in a secure form. A great example of this is SSL, or Secure Sockets Layer, which functions at Layer 6. However, IPSec is a set of standards that facilitate security at the Network Layer (Layer 3). It has two primary goals:
  • Protect all “selected” IP packets from snooping eyes
  • Defend against network attacks

IPSec accomplishes these goals by simply providing an “end-to-end” security model. What does that mean? Essentially, only the computers that are communicating need to know about the IPSec settings. No intermediate systems need to be aware of IPSec security, generally speaking. The destination and source nodes are responsible for handling security at their respective ends, based on the premise that the network their communication is occurring over is not secure.

Basically, IPSec attempts to make the entire network secure for all traffic rather than individual types of traffic. Since it works at the Network Layer, it can safeguard most protocols from higher levels—SMTP, HTTP, FTP, TCP, UDP, ICMP, etc. By securing all IP traffic at the lower levels of the OSI model, you provide the ultimate in secure communication and potentially secure all of the Internet and intranet traffic using the IPSec connection.

Many software companies that have realized the potential of IPSec are developing products that can use these standards. Cisco and Microsoft have been two of the forerunners of this surge, and both companies have support for IPSec in their latest product releases. Of course, we're going to focus here on IPSec in Win2K.

A look at IPSec in action
So how does this IPSec thing really work? Well, there's no short answer to that question, mostly because multiple pieces of technology make IPSec work, and an entire book could be written about it. But I'll try to summarize the basic idea of what is going on behind the scenes.

IPSec provides protection for traffic in three major areas:
  • Authentication
  • Integrity of the message
  • Confidentiality of the message

One of the key objectives of IPSec is to guarantee message identity. The “prove to me who you are” concept is extremely important to packet security. Verification of the origin of a message ensures that the packet actually did come from where it was supposed to. Hence, the authentication part of IPSec is meant to ensure the identity of the sender. IPSec provides authentication through one of three methods:
  • Public/private key cryptography with digital signatures. This is probably the most widely accepted method and, in my opinion, the recommended method. Basically, the sender and receiver will be issued certificates containing a public key that is associated with their respective private key. (A certificate authority, or CA, such as VeriSign, hands these out.) When a message is sent from one user to another, the sender will digitally sign the message using his or her private key, which is mathematically related to the individual's public key, by creating a hash that is sent with the packet. Upon receipt of the message, the receiving node will use the sender’s public key to derive a new hash that must match the one performed by the sender. If things match, the message has not changed, and you are who you say you are. If, for some reason, the hashes do not match, the source is not valid and cannot be authenticated, and the communication is rejected.
  • Kerberos. Authentication using this method works based on information provided to the participants of communication sessions from a Kerberos server. Kerberos is an industry standard method for authenticating a request for resource use. While Microsoft certainly did not invent this standard (it's attributed to some geeks at MIT), it has implemented it in Windows 2000 with Active Directory.
  • Preshared key. This method is pretty simplistic but nonetheless effective. Each side of the communication session predetermines (or shares) a key, which, in this case, is simply a text string. If computer 1 and computer 2 both decide to use the text string soccer as their preshared key, they can authenticate to one another. This technique is based on the concept that if the parties are the only ones who know the key, the entity trying to authenticate must be who it claims to be. This method, while secure, is not scalable. You'd have to manually enter this key into a policy, potentially numerous times, which would be labor-intensive. Also note that the key is stored in clear text, so it is not the most secure solution.

Integrity of the message
The integrity of the message, or ensuring that it does not change en route, is the job of the Authentication Header (AH). Slipped in behind the original IP packet's header, the AH is responsible for guarantees that the message was not altered while in transit. The diagram in Figure A shows an IP packet with the AH included.

Figure A

It is important to note that the AH does not provide confidentiality of the message; it only guarantees that it did not change during the communication process. The information contained within the packet could still be read, but it can't be altered. A process known as the HMAC, or Hash Message Authentication Codes, accomplishes integrity. The sender of a packet uses HMAC to hash or sign the IP packet with the public key of the sender. This hash value is included in the packet for use on the other end. Upon receipt of the packet, the receiver will perform a checksum on the packet and recompute the hash value. If the value is different, the packet has changed and is therefore rejected. If the numbers are the same, the packet has not been altered and is accepted as a good packet. Currently, IPSec supports the HMAC-MD5 and HMAC-SHA standards for message integrity.

Confidentiality of the message
Encrypting the message in transit so that itcannot be viewed is also a very important part of IPSec. The Encapsulating Security Payload (ESP) can be used to ensure confidentiality of the message. As was the case with the AH, the ESP header is inserted behind the normal IP header, making it possible to route through virtually any router without modification of the router. See the diagram in Figure B for an example of the ESP header.

Figure B

Encryption of IP packets is provided through these common security protocols:
  • Data Encryption Standard (DES). This uses a 56-bit key for its encryption and is widely accepted. Windows 2000’s implementation of IPSec supports 3DES, or three 56-bit keys.
  • Cipher block chaining (CBC). CBC is also supported for IPSec.

Do I need IPSec?
IPSec can be a valuable asset in your continual fight against security violations. Just think about the many types of attacks that administrators have to fend off:
  • Packet sniffing
  • Data modification
  • Address spoofing
  • Man-in-the-middle
  • Denial of service (DoS)

IPSec was designed with these problems in mind and has a number of key features that can greatly reduce your chances of getting slammed with one of these attacks. For example, address spoofing—someone attempting to impersonate an IP packet—can be prevented with the use of the AH, which ensures message integrity. The password compromise problem is solved with the authentication methods described earlier. Packet sniffing can be avoided with the use of ESP and its encryption schemes.

It is worth noting that while IPSec works with nearly all situations, IPSec has difficultly dealing with one glaring issue: network address translation (NAT). Currently, IPSec does not support NAT and will not make it through a NAT. This is something that should be considered in your plans for IPSec implementation.

IPSec allows you to lower your risk, which in turn allows you to lower your costs, which makes everybody happy. Of course, it does not solve all security problems, and as is always the case, nothing is totally secure. But it is a valuable tool for securing sensitive communications on your Win2K LAN.

Making it happen
In my next article, I'll continue this discussion of IPSec. Now that you have a foundational knowledge, we can move on to Win2K’s implementation of IPSec and walk through the steps to get IPSec up and running on a Windows 2000 network.

About Jeremy Smith

Jeremy L. Smith, CISSP, is a cybersecurity and public safety professional who has worked with a variety of agencies to improve the security of their call centers and execute their public safety initiatives more effectively, including 911 call taking,...

Editor's Picks

Free Newsletters, In your Inbox