Connect securely to Windows Vista Remote Desktop

Windows Remote Desktop Protocol (RDP) hasn't always had the best reputation for security. But since FIPS (Federal Information Processing Standard) grade security was added to Windows Server 2003 SP1, Windows Remote Desktop security has improved immensely. Walk through the steps to implement FIPS-grade security whenever you use Remote Desktop to connect to a Windows Vista computer from a Windows XP or Vista client machine.

This article is also available as a PDF download and gallery.

In the past, Windows RDP (Remote Desktop Protocol) hasn't had the best reputation for security. But since FIPS (Federal Information Processing Standard) grade security was added to Windows Server 2003 SP1 (Service Pack 1), Windows Remote Desktop has improved immensely on the security front. Now that this enhanced RDP technology has been added to Windows Vista, it's well within reach of the everyday home user. In this article, I'm going to show you how to implement FIPS-grade security whenever you Remote Desktop to a Windows Vista computer from a Windows XP or Vista client machine.


Software requirements

RDP server (host computer) RDP client computer
Windows Server 2003 SP1 and above Windows Vista any version
Windows Vista Business Editions Windows XP + RDP 6.0 client*
Windows Vista Ultimate Edition Windows Server 2003 + RDP 6.0*

* Link to RDP 6.0 client downloads


Secure configuration for Vista RDP host

The first thing you need to do is edit the Group Policy Object by running gpedit.msc, as shown in Figure A.

Figure A

Once inside the Group Policy Editor, navigate to Computer Configuration, Administrative Templates, Windows Components, Terminal Services, and then Security, as shown in Figure B.

Figure B

Next, set the Encryption Level to High Level, as shown in Figure C.

Figure C

Set Require Secure RPC Communication to Enabled, as shown in Figure D.

Figure D

Set Require Use Of Specific Security Layer For Remote (RDP) Connections to SSL (TLS 1.0), as shown in Figure E.

Figure E

Now, you must move to a different GPO section, at Computer Configuration, Windows Settings, Security Settings, Local Policies, and then Security Options, as shown in Figure F.

Figure F

Enable FIPS mode, as shown in Figure G.

Figure G

Enable Remote Desktop from the System Properties Window, as shown in Figure H. Note that you're setting it to allow any RDP 6.0 client rather than locking it down to permit only Vista clients.

Figure H

Once all this is configured, you must refresh the Group Policy to implement the new settings without a reboot. You do that with a forced GPUpdate, as shown in Figure I.

Figure I

Figure J shows a successful update.

Figure J


Secure RDP 6.0 client configuration

Now it's time to launch the RDP client using the MSTSC command, as shown in Figure K. Windows 2003 and XP users must download and install RDP 6.0 clients, whereas Vista comes with the correct client. On XP, you also need to launch the Run command before you can issue the MSTSC command.

Figure K

Enter the name of the server, noting that this initial process should happen on the LAN first. For this example, we're going to an RDP host machine called "msi-p965," shown in Figure L. This is not a fully qualified name, and it will work only on the same subnet LAN for now. It's possible to enter a redirect entry into the local host file pointing to an IP or dynamic DNS address so that you can access "msi-p965" or whatever you call your machine from the public Internet. However, we'll leave that for a follow-up article. For now, we're talking about just the immediate LAN.

Figure L

Next, you have to expand out Options, as shown in Figure M.

Figure M

Set the display to your liking using the options shown in Figure N.

Figure N

On the Local Resources tab, shown in Figure O, you can specify whether you want sound, printers, or the Clipboard to work.

Figure O

On the Programs tab, shown in Figure P, you can specify any programs you want to launch upon connection.

Figure P

Now, you can specify how you want the remote desktop to look using the settings shown in Figure Q. The more features you add, the more bandwidth it takes.

Figure Q

On the Advanced tab, shown in Figure R, you can set the RDP client to warn you if the RDP server fails to prove its authenticity. The name of the game is that you don't want to accidentally hand over your user credentials to a hacker who might be intercepting your connection.

Figure R

Click Settings and configure the options as shown in Figure S. In this example, we're telling it not to use a TS Gateway server.

Figure S

After you click OK, be sure you go back to the General tab and click Save As to save your entire profile. Otherwise, you'll have to do this whole procedure again next time. You can save it to the desktop for easy access.

Now click Connect and you'll be prompted for your username and password, as shown in Figure T.

Figure T

The first time you connect, you'll see the authentication warning shown in Figure U telling you that the server's certificate is not trusted (yet). To rectify this situation and force it to be trusted in the future, click the View Certificate button.

Figure U

As you can see in Figure V, this self-signed cert generated by the Vista RDP host machine is valid for the next six months. Click on the Install Certificate button to add it to the CTL (Certificate Trust List).

Figure V

The Certificate Import Wizard will launch, as shown in Figure W. Click Next to proceed.

Figure W

Choose Place All Certificates In The Following Store and click the Browse button, shown in Figure X.

Figure X

Select Show Physical Stores and highlight Local Computer, as shown in Figure Y.

Figure Y

Back in the Certificate Store screen, shown in Figure Z, click Next.

Figure Z

To complete the import, just click the Finish button, shown in Figure AA.

Figure AA

When you see the success message shown in Figure AB, click OK.

Figure AB

At this point, you'll be securely connected to the Vista RDP host, but more important, future connections to msi-p965 won't result in any warning signs or even password prompts. It will simply connect in a secure manner, and any warning signs must be viewed with a critical eye.

What happens when you try to connect to this host via IP address or a dynamic DNS entry from the public Internet? If you try to connect by any name other than the one you used to originally generate the certificate (in this example, it's "msi-p965"), you will see a warning like the one shown in Figure AC. You can tell it to connect anyway and choose Don't Prompt Me Again For Connections To This Computer.

Figure AC

You'll then get another warning, like the one shown in Figure AD, that tells you there's a name mismatch and that the server name on the certificate is incorrect. This isn't a bad thing. You can view the certificate and it will say it's for "msi-p965" and that it's trusted. You're just seeing this warning because the RDP client is comparing the name on the certificate with the name of the computer you're connecting to. For this example, I was trying to connect to "" and not "msi-p965", so the computer warned me that they didn't match. Since I intended to connect to that IP address or some other publicly resolvable DNS name on the public Internet, and since the certificate was valid, I knew I wasn't being deceived. So I was comfortable clicking Yes to connect anyway. To avoid seeing this error in the future, I'll need to edit the local host file to map the IP or DNS name to "msi-p965" or whatever the name of my machine is.

Figure AD

But what if a hacker poses as your server with a made-up certificate? In that case, you'll see the warning shown in Figure AE telling you that not only does the name not match, the certificate isn't even from a trusted certifying authority. If you see this kind of error when you've already gone through the certificate installation procedure from Figure U to Figure AB, you know someone is trying to dupe you. You should click No and not connect to the server. If you attempt to make the connection anyway, you'll reveal enough of your credentials for the hacker to quickly run a dictionary attack to find your password.

Figure AE

If this seems like a rather complex process just to get no warning signs for an RDP connection, it is—but it's the only practical way to establish a secure and trusted connection. Fortunately, you have to do it only once, and all subsequent connections are secure and hassle free. Believe it or not, you've essentially created your own PKI certificate on the RDP host and installed a Certificate Authority on the client computer. This level of security using a Public Key Exchange is used to secure e-commerce transactions. On an enterprise level, this entire procedure with GPO settings and digital certificates can actually be automated on both the server and the client side using Active Directory Group Policies, but now you know how it all works.

In a future article, I'll show you how to set up a free dynamic DNS entry that's publicly resolvable and that points to your home dynamic IP broadband service. When everything is secure, we'll trick the client machine into not generating any more warning messages at all.

Editor's Picks