Networking

Configure IT Quick: Configure Windows 2000 IPSec to secure network traffic

Make a Windows 2000 network more secure


IPSec is fast becoming the de facto standard for secure communications on the Internet. However, did you know that you can also use it to secure sensitive transmissions on your LAN and/or WAN if you're running Windows 2000? My previous article introduced the Windows implementation of IPSec. Now, we'll walk through the process of setting up and configuring IPSec in Win2K.

Windows 2000 support
Microsoft has included significant support for IPSec in Windows 2000 (and Windows XP as well), and this implementation offers several key benefits:
  • The Windows 2000 IPSec implementation is easy to deploy. IPSec is fully integrated with Windows 2000 domain security, using it for its trust model. By default, an IPSec policy will use Kerberos v5 authentication, which is also built into a Windows 2000 domain. Thus, a computer that is a member of a Win2K domain can easily participate in secure communication using IPSec.
  • In Win2K, IPSec is transparent to users and applications. As we learned in my previous article, IPSec operates at Layer 3 of the OSI model, which allows for nearly any application to utilize it, effectively shielding the complexity and compatibility issues from the higher layers of the OSI model.
  • Win2K provides support for all aspects of IPSec, which makes it compatible with IPSec implementations from other vendors.

Let's take a look at how you can get IPSec running on your Win2K network.

How to set up and configure IPSec
The most effective method for implementing IPSec in a Windows 2000 environment is through the use of policies. To use IPSec policies, you must have a keen understanding of Microsoft’s new administrative tools interface, the Microsoft Management Console (MMC). If you're familiar with the MMC, you're ready to jump in. If not, you'll want to take a quick look at the Win2K help files first.

To begin, click Start | Run, type MMC, and click OK. This will open up a blank MMC console from which you need to click Console | Add/Remove Snap-in. Next, click Add | IP Security Policy Management. You will be asked to choose what Group Policy object you want to manage. Choose the Local Computer object and then click Finish, Close, and OK. You've just installed the administrative tool you will use to configure IPSec policies for your machine.

Once you are back in the console, click IP Security Policies On Local Machine in the left pane. You will notice that three policies are already defined for you:
  • Secure Server (Require Security)
  • Client (Respond Only)
  • Server (Request Security)

Notice, too, that under the Policy Assigned column, each policy is set to No (unless you or someone else changed them beforehand). In a minute, I will explain the ramifications of each default policy.

First, you need to know that IPSec policies consist of six major components. You can access these components by right-clicking on any of the three default policies and selecting Properties. The components are as follows:
  • Rules—A rule controls how and when an IPSec policy will govern secure communication. A rule will be applied whenever a computer has a positive match with the criteria specified on the Filter List. You can see your rules by selecting a policy, viewing its Properties, and looking at the Rules tab.
  • Filter lists—A filter list, which can be viewed by selecting a rule and clicking the Edit button, allows you to see the source and destination IP addresses the selected rule applies to. Filter lists can control inbound and outbound traffic. You will see the Filter Lists on the Rules tab.
  • Filter actions—Once a computer has met the specified criteria on a rule’s filter list, it will have filter actions applied to it. With filter actions, you have three possible outcomes for the communication attempt: Permit, Request, and Require. Filter actions can be viewed from the Filter Action tab when you go into the Edit function of a rule.
  • Connection type—The connection type determines the connections that will be affected by this IPSec policy. For example, you may have a LAN connection and a VPN connection and would like only the VPN connection to be affected by the IPSec policy. The connection type lets you make those changes. You can set the type by selecting the Connection Type tab when you go into the Edit function of a rule.
  • Authentication—This is the method of authentication the selected policy will employ as it attempts to establish communication with another node on the network. The three authentication methods are Kerberos v5, Preshared key, and public key certificate. You can set this by clicking on the Authentication Method tab when you go into the Edit function of a rule.
  • Tunnel setting—IPSec can run in one of two modes: Transport Mode or Tunnel Mode. This setting lets you decide which mode you want this policy to be in. Transport Mode is most commonly used for normal LAN communication, whereas Tunnel Mode might be used to set up a secure tunnel between two routers connecting a couple of remote offices. Incidentally, Tunnel Mode is used by L2TP VPNs. You access this setting by going to the Tunnel Setting tab on the Edit function of a rule. By choosing the This Rule Does Not Specify An IPSec Tunnel option, you put the policy into Transport Mode. To use Tunnel Mode, choose The Tunnel Endpoint Is Specified By This IP Address and then enter in the IP address of the other end of the tunnel.

Default policies
Let’s take a closer look at the default policies. As I mentioned before, there are three policies provided for you by default:
  • Client (Respond Only)—By applying this policy, you are deciding to set your computer in respond-only mode. That means that if another computer were to ask your computer to communicate using IPSec, it could respond positively and communication would be secured. However, it will never initiate secure communications—it will only respond. This setting would be the most compatible setting and perhaps the most common setting clients would use.
  • Server (Request Security)—This policy should be assigned to computers that need to communicate securely most of the time. A client that has been assigned this policy will always “ask” to communicate securely, and if the destination supports it, communication will be secured. If the destination does not support IPSec, normal, unsecured communication can take place.
  • Secure Server (Require Security)—This policy should be applied only to computers that require the utmost in secure communications. With this policy, computers must support IPSec to communicate. If a system doesn’t support it, communication will fail. The policy allows unsecured incoming traffic, but outgoing traffic must be secured with IPSec.

Now that you understand the ramifications of each of the default policies, let's see how to apply one of them to a computer. In your IPSec MMC console, select one of the default policies, right-click on it, and choose Assign. You will now notice that the Policy Assigned column displays Yes next to the policy you just assigned. Keep in mind that you can have only one policy assigned at any time.

You'll also need to start the IPSec Policy Agent service to put your assigned policy into effect. To do this, go to Start | Programs | Administrative Tools | Computer Management and then expand Services And Applications and click on Services. Right-click on IPSec Policy Agent and select Properties. Click the Start button and then change the Startup Type to Automatic and click OK.

Custom policies
In addition to the aforementioned default policies, you can create custom policies to meet specific security needs not addressed by a default policy. To create a custom policy, right-click on IP Security Policies On Local Machine (in the left pane) and select Create IP Security Policy. This will launch a wizard that will guide you through the process of creating a customized policy for this machine. The wizard is fairly self-explanatory, allowing you to choose the appropriate encryption levels and authentication mechanisms for your situation. Custom policies can be an effective method of implementing an IPSec solution.

Final thoughts
In this article, we set up IPSec using a local policy; however, I recommend using group policies as a centralized method of controlling IPSec rather than going to each Windows 2000 system and setting a local policy. However, the process is essentially the same, and the setup that you learned in this article can easily be applied to setting up a group policy.

Setting up and configuring IPSec is really not that hard. Perhaps the more difficult task is understanding the many intricacies of IPSec itself. In my opinion, Microsoft has done a decent job of allowing a network administrator to successfully implement and manage IPSec policies and has developed a highly secure solution that can positively affect your network's security.

Editor's Picks

Free Newsletters, In your Inbox