This article is also available as a PDF download and gallery.

In the past, Windows RDP (Remote Desktop Protocol) hasn’t had the best
reputation for security. But since

FIPS (Federal Information Processing Standard) grade security was
added to Windows Server 2003 SP1 (Service Pack 1), Windows Remote Desktop has
improved immensely on the security front. Now that this enhanced RDP
technology has been added to Windows Vista, it’s well within reach of the everyday home user. In this article, I’m going to show you how to implement
FIPS-grade security whenever you Remote Desktop to a Windows Vista computer from a Windows XP or Vista client machine.

 

Software requirements

RDP server (host computer) RDP client computer
Windows Server 2003 SP1 and above Windows Vista any version
Windows Vista Business Editions Windows XP + RDP 6.0 client*
Windows Vista Ultimate Edition Windows Server 2003 + RDP 6.0*

* Link to RDP 6.0 client
downloads

 

Secure configuration for Vista RDP host

The first thing you need to do is edit the Group Policy Object by running gpedit.msc, as shown in Figure A.

Figure A

Once inside the Group Policy Editor, navigate to Computer
Configuration, Administrative Templates, Windows Components, Terminal Services,
and then Security, as shown in Figure B.

Figure B

Next, set the Encryption Level to High Level, as shown in Figure C.

Figure C

Set Require Secure RPC Communication to Enabled, as shown in Figure D.

Figure D

Set Require Use Of Specific Security Layer For Remote (RDP)
Connections to SSL (TLS 1.0), as shown in Figure E.

Figure E

Now, you must move to a different GPO section, at Computer Configuration,
Windows Settings, Security Settings, Local Policies, and then Security Options, as
shown in Figure F.

Figure F

Enable FIPS mode, as shown in Figure G.

Figure G

Enable Remote Desktop from the System Properties Window, as shown
in Figure H. Note that you’re setting it to allow any RDP 6.0 client
rather than locking it down to permit only Vista clients.

Figure H

Once all this is configured, you must refresh the Group Policy to implement
the new settings without a reboot. You do that with a forced GPUpdate, as
shown in Figure I.

Figure I

Figure J shows a successful update.

Figure J

 

Secure RDP 6.0 client configuration

Now it’s time to launch the RDP client using the MSTSC command, as shown
in Figure K. Windows 2003 and XP users must
download and install
RDP 6.0 clients, whereas Vista comes with the correct client. On XP, you
also need to launch the Run command before you can issue the MSTSC command.

Figure K

Enter the name of the server, noting that this initial
process should happen on the LAN first. For this example, we’re going to an RDP
host machine called “msi-p965,” shown in Figure L. This is not a
fully qualified name, and it will work only on the same subnet LAN for now. It’s possible to enter a redirect entry into the local host file
pointing to an IP or dynamic DNS address so that you can access “msi-p965” or
whatever you call your machine from the public Internet. However, we’ll leave that
for a follow-up article. For now, we’re talking about just the immediate LAN.

Figure L

Next, you have to expand out Options, as shown in Figure M.

Figure M

Set the display to your liking using the options shown in Figure N.

Figure N

On the Local Resources tab, shown in Figure O, you can specify whether you want sound, printers, or the Clipboard to work.

Figure O

On the Programs tab, shown in Figure P, you can specify any programs you want to launch upon connection.

Figure P

Now, you can specify how you want the remote desktop to look using the settings shown in Figure Q. The
more features you add, the more bandwidth it takes.

Figure Q

On the Advanced tab, shown in Figure R, you can set the RDP client to warn you if the RDP server fails to prove its
authenticity. The name of the game is that you don’t
want to accidentally hand over your user credentials to a hacker who might be
intercepting your connection.

Figure R

Click Settings and configure the options as shown in Figure S.
In this example, we’re telling it not to use a TS Gateway server.

Figure S

After you click OK, be sure you go back to the General tab and click Save As to save your entire profile. Otherwise, you’ll have
to do this whole procedure again next time. You can save it to the desktop
for easy access.

Now click Connect and you’ll be prompted for your username and
password, as shown in Figure T.

Figure T

The first time you connect, you’ll see the authentication warning shown
in Figure U telling you that the server’s certificate is not trusted (yet). To rectify this situation and force it to be trusted in the
future, click the View Certificate button.

Figure U

As you can see in Figure V, this self-signed cert generated by the Vista RDP
host machine is valid for the next six months. Click on the Install
Certificate button to add it to the CTL (Certificate Trust List).

Figure V

The Certificate Import Wizard will launch, as shown in Figure W. Click Next to proceed.

Figure W

Choose Place All Certificates In The Following Store and click the Browse button, shown in Figure X.

Figure X

Select Show Physical Stores and highlight Local Computer, as shown
in Figure Y.

Figure Y

Back in the Certificate Store screen, shown in Figure Z, click Next.

Figure Z

To complete the import, just click the Finish button, shown in Figure AA.

Figure AA

When you see the success message shown in Figure AB, click OK.

Figure AB

At this point, you’ll be securely connected to the Vista RDP host, but more
important, future connections to msi-p965 won’t result in any
warning signs or even password prompts. It will simply connect in
a secure manner, and any warning signs must be viewed with a critical eye.

What happens when you try to connect to this host via IP address or a
dynamic DNS entry from the public Internet? If you try to connect by any
name other than the one you used to originally generate the certificate (in this example, it’s “msi-p965”), you will see a warning like the one shown in Figure AC.
You can tell it to connect anyway and choose Don’t Prompt Me Again For Connections To
This Computer.

Figure AC

You’ll then get another warning, like the one shown in Figure AD, that tells you there’s a name mismatch and that the server name on the certificate is incorrect. This isn’t a bad thing. You can view the certificate and it will say it’s for “msi-p965” and that it’s trusted. You’re just seeing this warning because the RDP client is comparing the name on the certificate with the name of
the computer you’re connecting to. For this example, I was trying to connect to
“192.168.1.2” and not “msi-p965”, so the computer warned me that they didn’t match.
Since I intended to connect to that IP address or some
other publicly resolvable DNS name on the public Internet, and since the certificate was valid, I knew I wasn’t being deceived. So I was comfortable clicking Yes to connect anyway. To avoid seeing this error in the future, I’ll need to edit the local host file to map the IP or DNS name to “msi-p965” or whatever the name of my machine is.

Figure AD

But what if a hacker poses as your server with a made-up
certificate? In that case, you’ll see the warning shown in Figure AE telling you
that not only does the name not match, the certificate isn’t even from a trusted
certifying authority. If you see this kind of error when you’ve already gone
through the certificate installation procedure from Figure U to Figure AB, you know someone is trying to dupe you. You should click No and not connect to
the server. If you attempt to make the connection anyway, you’ll reveal
enough of your credentials for the hacker to quickly run a dictionary attack to
find your password.

Figure AE

If this seems like a rather complex process just to get no warning signs for an RDP
connection, it is–but it’s the only practical way to establish a secure and
trusted connection. Fortunately, you have to do it only once, and all
subsequent connections are secure and hassle free. Believe it or not,
you’ve essentially created your own PKI certificate on the RDP host and
installed a Certificate Authority on the client computer. This level of
security using a Public Key Exchange is used to secure e-commerce transactions. On an enterprise level, this entire procedure with GPO settings and digital
certificates can actually be automated on both the server and the client side
using Active Directory Group Policies, but now you know how it all works.

In a future article, I’ll show you how to set up a free dynamic DNS
entry that’s publicly resolvable and that points to your home dynamic IP broadband
service. When everything is secure, we’ll trick the client machine into not
generating any more warning messages at all.