Microsoft

SolutionBase: Protect your workstation with Windows XP SP2's Data Execution Prevention technology

With all of the viruses, spyware, and other exploits running around the Internet, wouldn't it be nice if the computer could protect itself a little bit? Using XP SP2's DEP feature, it just might be able to. Greg Shultz explains how it works.

In addition to antivirus and firewall protection, a relatively new security technology called Data Execution Prevention, or DEP for short, is available to protect your computer from malicious code. DEP, which was introduced with Windows XP SP2, works by monitoring memory usage for malicious code that exploits vulnerabilities in memory heaps, stacks, and pools. Unencumbered, these types of attacks, such as the recent WMF (Windows Meta File) vulnerability or Code Red Buffer Overflow, can bring a system to its knees in no time flat. Other types of similar attacks can use these vulnerabilities to turn a system into a springboard to propagate the malicious code out across the network as well as to other local programs.

To combat these types of attacks Microsoft and the major CPU manufacturers, AMD and Intel, worked together to develop the DEP technology and integrate DEP components into both the CPU and the operating system. The hardware-enforced component of DEP is built into the new AMD and Intel processors and the software-enforced component of DEP is built into Windows XP SP2.

However, it's important to keep in mind that while these two DEP components have the same goal and can indeed work together, they are separate and for compatibility reasons aren't simultaneously enabled by default. In fact, in its default configuration, only the software-enforced DEP component is enabled while the hardware-enforced DEP component is disabled. Fortunately, Windows XP SP2 allows you to tap the software-enforced DEP component into the hardware-enforced DEP component to combine them and enhance overall system protection.

In this article, I'll examine both the hardware-enforced and software-enforced components of DEP and explain how they work. I'll then show you how to find and configure Windows XP SP2 to use both software- and hardware-enforced DEP for maximum protection. As I do, I'll show you how to use the built-in exception mechanism to work around any incompatibilities you might encounter with your applications.

DEP in general

To begin with, let's take a closer look at DEP in general. As I alluded to in the introduction, DEP is designed to add an extra level of protection to your system in addition to that offered by antivirus and firewall programs. The thing to keep in mind is that in contrast to antivirus and firewall programs, hardware and software-enforced DEP technologies are not designed to prevent malicious programs from being installed on your computer.

Rather, DEP as a whole is designed to monitor running programs to ensure that they are safely using system memory. As such, not only can DEP protect your system from malicious programs that have the potential to evade detection by antivirus and firewall programs, but DEP can also protect your system from safe programs that may occasionally run amuck due to some abnormal situation.

Hardware-enforced DEP

As its name implies, hardware-enforced DEP is based on logic built into the processor and is designed to monitor all application code as it is being loaded into memory. More specifically, hardware-enforced DEP keeps track of each memory location that an application uses and marks those locations that do not specifically contain executable code with a special attribute or flag. With those flags set, the application is free to perform its tasks and the processor goes about its business as it normally would.

However, if malicious code tries to sneak an unauthorized executable operation into an available, but flagged memory location, the processor raises an exception. When this occurs, the malicious code is intercepted and rejected and the potential attack is averted.

This same series of events is also triggered if an authorized program goes awry and inadvertently attempts access a memory location not available for executable code. In addition to a program going awry, there are some legitimate programs, typically legacy or poorly written programs, that may attempt use a memory location not available for executable code. In both of these cases, the program would be prevented from running any further and would fail--usually resulting in the Blue Screen of Death.

As I mentioned in the introduction, both the newer AMD and Intel processors have hardware-enforced DEP built into them. More specifically, the hardware-enforced DEP in AMD processors is called the no-execute page-protection (NX) processor feature while the hardware-enforced DEP in Intel processors is called the Execute Disable bit feature. Regardless of their different names, both architectures are Windows compatible and can work with the software-enforced DEP features built into Windows XP SP2.

Software-enforced DEP

Windows XP SP2's software-enforced DEP works similarly to the hardware-enforced DEP, in that it monitors memory locations for unauthorized access. However, SP2's software-enforced DEP is only configured to monitor the memory locations used by crucial operating system executables and services. By default, the hardware-enforced DEP component, which is accessible from Windows XP SP2, is not enabled.

This intended hobbling of the DEP technology was done in the name of compatibility in order to allow those legitimate legacy programs that don't adhere to the DEP code of behavior to run unimpeded by errors and failures. In other words, disabling hardware-enforced DEP helps to dispel the misguided belief that Windows XP SP2 breaks applications.

Checking hardware-enforced DEP status

If you're not sure whether the processor in a computer running Windows XP SP2 is capable of providing hardware-enforced DEP, you can easily find out by accessing Data Execution Prevention configuration tool in Windows XP SP2. To do so, press [Windows]+[Break] to bring up the System Properties dialog box. Next, select the Advanced tab and then click the Settings button in the Performance section. When the Performance Options dialog box appears, select the Data Execution Prevention tab.

If the computer is not capable of providing hardware-enforced DEP, you'll see a notice at the bottom of the dialog box like the one shown in Figure A. In this case, the computer is only using SP2's software-enforced DEP.

Figure A

If the computer isn't capable of providing hardware-enforced DEP, you'll see a notice to that effect at the bottom of the Data Execution Prevention tab.

On the other hand, if the computer is capable of providing hardware-enforced DEP, you won't see anything at the bottom of the dialog box, as shown in Figure B. However, you'll notice that even though the computer is capable of providing hardware-enforced DEP, Windows XP is only configured to use software-enforced DEP.

Figure B

Even if the computer is capable of providing hardware-enforced DEP, Windows XP is only configured to use software-enforced DEP.

Enabling hardware-enforced DEP

If the processor in a computer running Windows XP SP2 is capable of providing hardware-enforced DEP, you definitely want to enable it. Doing so may actually save your system someday. For instance, during the recent outbreak of the WMF exploit, many fellow techs reported that software-enforced DEP alone was unable to thwart the exploit, but that hardware-enforced DEP, when enabled, was able to protect systems. (Keep in mind that regardless of whether hardware-enforced DEP offered protection from the WMF exploit, you should visit Windows/Microsoft Update and get the patch!)

Fortunately, enabling hardware-enforced DEP is easy. To do so, just access the Data Execution Prevention tab and select the Turn On DEP For All Programs And Services Except Those I Select option, as shown in Figure C.

Figure C

Enabling hardware-enforced DEP is a simple procedure.

To complete the operation, click OK. When you do, you'll see the dialog box in Figure D and will again click OK. You'll then need to close all open applications and then manually restart Windows XP.

Figure D

Once you enable hardware-enforced DEP, you'll be prompted to manually restart your system.

Working with DEP exceptions

As I mentioned earlier, hardware-enforced DEP is not enabled in Windows XP SP2 by default because it would often prevent legitimate legacy programs that don't adhere to the DEP code of behavior from running, thus giving the impression that the security features in SP2 would break applications. Apparently there are a lot of legacy programs that don't adhere to the DEP code of behavior.

As such, after enabling hardware-enforced DEP from the Data Execution Prevention tab, you may indeed encounter a problem between DEP and a legacy program. If you do, you'll encounter a Data Execution Prevention warning dialog box, like the one shown in Figure E.

Figure E

If you have an application that doesn't adhere to the DEP code of behavior, you'll encounter a Data Execution Prevention warning dialog box.

Fortunately, this is not a show stopper as you can add applications to the DEP exception list. To do so, just access the Data Execution Prevention tab and click the Add button towards the bottom of the dialog box. When you do so, you'll see an Open dialog box, like the one in Figure F, and can locate and select the application's executable file.

Figure F

You can easily add applications to the DEP exception list.

Of course, Microsoft recommends that you contact the application vendor to determine if a DEP-compatible update is available. Microsoft then goes on to say that installing such an update is the preferred solution for DEP compatibility issues.

Configuring DEP via Boot.ini

In addition to configuring DEP on the Data Execution Prevention tab, you can also make changes by editing the operating system line in the Boot.ini file. To do so, you modify the /noexecute switch with one of the available policy levels listed in Table A, as shown here:

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft
Windows XP Professional" /fastdetect /noexecute = Policy_level

Protection built in

Data Execution Prevention or DEP is designed to protect a computer from malicious code by monitoring memory for unauthorized executable code. DEP consists of two components: software-enforced DEP and hardware-enforced DEP. The software component of DEP is built into Windows XP SP2 while the hardware component of DEP is built into the newer AMD and Intel processors. By default, Windows XP SP2 only enables software-enforced DEP and requires that you manually enable and hardware-enforced DEP.

About

Greg Shultz is a freelance Technical Writer. Previously, he has worked as Documentation Specialist in the software industry, a Technical Support Specialist in educational industry, and a Technical Journalist in the computer publishing industry.

0 comments

Editor's Picks