Windows

Connect securely to Windows Vista Remote Desktop

Windows Remote Desktop Protocol (RDP) hasn't always had the best reputation for security. But since FIPS (Federal Information Processing Standard) grade security was added to Windows Server 2003 SP1, Windows Remote Desktop security has improved immensely. Walk through the steps to implement FIPS-grade security whenever you use Remote Desktop to connect to a Windows Vista computer from a Windows XP or Vista client machine.

This article is also available as a PDF download and gallery.

In the past, Windows RDP (Remote Desktop Protocol) hasn't had the best reputation for security. But since FIPS (Federal Information Processing Standard) grade security was added to Windows Server 2003 SP1 (Service Pack 1), Windows Remote Desktop has improved immensely on the security front. Now that this enhanced RDP technology has been added to Windows Vista, it's well within reach of the everyday home user. In this article, I'm going to show you how to implement FIPS-grade security whenever you Remote Desktop to a Windows Vista computer from a Windows XP or Vista client machine.

 

Software requirements

RDP server (host computer) RDP client computer
Windows Server 2003 SP1 and above Windows Vista any version
Windows Vista Business Editions Windows XP + RDP 6.0 client*
Windows Vista Ultimate Edition Windows Server 2003 + RDP 6.0*

* Link to RDP 6.0 client downloads

 

Secure configuration for Vista RDP host

The first thing you need to do is edit the Group Policy Object by running gpedit.msc, as shown in Figure A.

Figure A

Once inside the Group Policy Editor, navigate to Computer Configuration, Administrative Templates, Windows Components, Terminal Services, and then Security, as shown in Figure B.

Figure B

Next, set the Encryption Level to High Level, as shown in Figure C.

Figure C

Set Require Secure RPC Communication to Enabled, as shown in Figure D.

Figure D

Set Require Use Of Specific Security Layer For Remote (RDP) Connections to SSL (TLS 1.0), as shown in Figure E.

Figure E

Now, you must move to a different GPO section, at Computer Configuration, Windows Settings, Security Settings, Local Policies, and then Security Options, as shown in Figure F.

Figure F

Enable FIPS mode, as shown in Figure G.

Figure G

Enable Remote Desktop from the System Properties Window, as shown in Figure H. Note that you're setting it to allow any RDP 6.0 client rather than locking it down to permit only Vista clients.

Figure H

Once all this is configured, you must refresh the Group Policy to implement the new settings without a reboot. You do that with a forced GPUpdate, as shown in Figure I.

Figure I

Figure J shows a successful update.

Figure J

 

Secure RDP 6.0 client configuration

Now it's time to launch the RDP client using the MSTSC command, as shown in Figure K. Windows 2003 and XP users must download and install RDP 6.0 clients, whereas Vista comes with the correct client. On XP, you also need to launch the Run command before you can issue the MSTSC command.

Figure K

Enter the name of the server, noting that this initial process should happen on the LAN first. For this example, we're going to an RDP host machine called "msi-p965," shown in Figure L. This is not a fully qualified name, and it will work only on the same subnet LAN for now. It's possible to enter a redirect entry into the local host file pointing to an IP or dynamic DNS address so that you can access "msi-p965" or whatever you call your machine from the public Internet. However, we'll leave that for a follow-up article. For now, we're talking about just the immediate LAN.

Figure L

Next, you have to expand out Options, as shown in Figure M.

Figure M

Set the display to your liking using the options shown in Figure N.

Figure N

On the Local Resources tab, shown in Figure O, you can specify whether you want sound, printers, or the Clipboard to work.

Figure O

On the Programs tab, shown in Figure P, you can specify any programs you want to launch upon connection.

Figure P

Now, you can specify how you want the remote desktop to look using the settings shown in Figure Q. The more features you add, the more bandwidth it takes.

Figure Q

On the Advanced tab, shown in Figure R, you can set the RDP client to warn you if the RDP server fails to prove its authenticity. The name of the game is that you don't want to accidentally hand over your user credentials to a hacker who might be intercepting your connection.

Figure R

Click Settings and configure the options as shown in Figure S. In this example, we're telling it not to use a TS Gateway server.

Figure S

After you click OK, be sure you go back to the General tab and click Save As to save your entire profile. Otherwise, you'll have to do this whole procedure again next time. You can save it to the desktop for easy access.

Now click Connect and you'll be prompted for your username and password, as shown in Figure T.

Figure T

The first time you connect, you'll see the authentication warning shown in Figure U telling you that the server's certificate is not trusted (yet). To rectify this situation and force it to be trusted in the future, click the View Certificate button.

Figure U

As you can see in Figure V, this self-signed cert generated by the Vista RDP host machine is valid for the next six months. Click on the Install Certificate button to add it to the CTL (Certificate Trust List).

Figure V

The Certificate Import Wizard will launch, as shown in Figure W. Click Next to proceed.

Figure W

Choose Place All Certificates In The Following Store and click the Browse button, shown in Figure X.

Figure X

Select Show Physical Stores and highlight Local Computer, as shown in Figure Y.

Figure Y

Back in the Certificate Store screen, shown in Figure Z, click Next.

Figure Z

To complete the import, just click the Finish button, shown in Figure AA.

Figure AA

When you see the success message shown in Figure AB, click OK.

Figure AB

At this point, you'll be securely connected to the Vista RDP host, but more important, future connections to msi-p965 won't result in any warning signs or even password prompts. It will simply connect in a secure manner, and any warning signs must be viewed with a critical eye.

What happens when you try to connect to this host via IP address or a dynamic DNS entry from the public Internet? If you try to connect by any name other than the one you used to originally generate the certificate (in this example, it's "msi-p965"), you will see a warning like the one shown in Figure AC. You can tell it to connect anyway and choose Don't Prompt Me Again For Connections To This Computer.

Figure AC

You'll then get another warning, like the one shown in Figure AD, that tells you there's a name mismatch and that the server name on the certificate is incorrect. This isn't a bad thing. You can view the certificate and it will say it's for "msi-p965" and that it's trusted. You're just seeing this warning because the RDP client is comparing the name on the certificate with the name of the computer you're connecting to. For this example, I was trying to connect to "192.168.1.2" and not "msi-p965", so the computer warned me that they didn't match. Since I intended to connect to that IP address or some other publicly resolvable DNS name on the public Internet, and since the certificate was valid, I knew I wasn't being deceived. So I was comfortable clicking Yes to connect anyway. To avoid seeing this error in the future, I'll need to edit the local host file to map the IP or DNS name to "msi-p965" or whatever the name of my machine is.

Figure AD

But what if a hacker poses as your server with a made-up certificate? In that case, you'll see the warning shown in Figure AE telling you that not only does the name not match, the certificate isn't even from a trusted certifying authority. If you see this kind of error when you've already gone through the certificate installation procedure from Figure U to Figure AB, you know someone is trying to dupe you. You should click No and not connect to the server. If you attempt to make the connection anyway, you'll reveal enough of your credentials for the hacker to quickly run a dictionary attack to find your password.

Figure AE

If this seems like a rather complex process just to get no warning signs for an RDP connection, it is--but it's the only practical way to establish a secure and trusted connection. Fortunately, you have to do it only once, and all subsequent connections are secure and hassle free. Believe it or not, you've essentially created your own PKI certificate on the RDP host and installed a Certificate Authority on the client computer. This level of security using a Public Key Exchange is used to secure e-commerce transactions. On an enterprise level, this entire procedure with GPO settings and digital certificates can actually be automated on both the server and the client side using Active Directory Group Policies, but now you know how it all works.

In a future article, I'll show you how to set up a free dynamic DNS entry that's publicly resolvable and that points to your home dynamic IP broadband service. When everything is secure, we'll trick the client machine into not generating any more warning messages at all.

21 comments
alokgovil
alokgovil

Hi George, I had followed these steps to successfully configure RDP. As you noted, the certificate expires every six months. The issue I am having while renewing it is that in Figure Y, I do not see "Local Computer" at all. I have tried both normal-user and administrator accounts for both machines. Please help! Thanks, Alok

sfun28
sfun28

Has this been updated for Win7?

wilkinson.john
wilkinson.john

After setting all the settings shown here, I did a GPUPDATE /target:computer and then restarted my PC for good messure but when I run RSOP.MSC it still shows the "System cryptography: Use FIPS compliant algorithms for encryption, hashing and signing policy" still shows as disabled, any one else having the same problem?

chacko_saju
chacko_saju

Can someone please explain how to connect win2k3 sp2 server using RDP 6.0 along with smartcard certification authentication from a WinXP client? Assuming I already have install RDP 6.0 client on my winxp system and Enterprise root CA is installed on the same W2k3 server to which i'm making a rdp connection. Thanx in advance SC

cab
cab

Article was useful BUT I found out the hard way that when you enable FIPS, the Microsoft Windows Home Server will stop backing up the computer. Have not heard from MS if/when they will support FIPS for the just released WHS, but you might make a note of this.

cab
cab

Article was useful BUT I found out the hard way that when you enable FIPS, the Microsoft Windows Home Server will stop backing up the computer. Have not heard from MS if/when they will support FIPS for the just released WHS, but you might make a note of this.

newalloy
newalloy

A very important note about enabling FIPS compliance here: Doing so will cause some https websites to not function in IE. The reason is that the server that the website resides on might not support FIPS. IIS based servers will, but some others that use SSL 3.0 might not... For instance, my vonage account won't load in IE, nor will my cable companies online billing. They don't use FIPS compliant algorithms.

JodyGilbert
JodyGilbert

How well does this technique work for the Remote Desktop demands of your environment? Does it overcome security concerns that have been an obstacle for you in the past?

alokgovil
alokgovil

Hi again, I figured. Logging in as administrator (using Windows 7) is not enough. I had to right-click RDP and choose "Run as administrator". Thanks, Alok

anona678
anona678

How do you configure the SSL to be used when you connect via RDP in Vista as in the server OS you have that opinion in "Terminal Services Configuration"?

mike
mike

Have they made this any easier? There are few documents to assist in helping with remote printing off a RDT connection

TelcoChuck
TelcoChuck

This article is very good for use on my SBS2003 machine, but what about the XP machines ? will this enhanced security be migrated to XP Pro? Have our friends in Redmond given any hints?

georgeou
georgeou

you just check off a few check marks and it should work.

georgeou
georgeou

You can't use this on XP on the host/server side, but you can use XP on the client side as shown in article. Microsoft will not be adding more features to XP other than security updates. They need to be able to make a case for moving to Vista.

uberg33k50
uberg33k50

I have users who work from home in the evening. I have a obscure port number routed to each of their workstations through the firewall. If I could put this security on their workstation and then setup their home computer for the security settings that would be awesome.

mike
mike

I just returned from the TS2 session in Las Vegas on server 2008. It appears they have made printing as easy as using Simplified Printing. For all of you stating it is as easy as placing a check in the Printer" checkbox in resources, I have not found that to be true at all. I still use Tricerats Simplified Printing and cannot wait for the release of svr 2oo8. It also provides the ability to RDP the application instead of the whole desktop. This is great for certain applications but I will need to test it with aps that aren't designed to work with Remote Desktop"

travisn000
travisn000

It is just as easy as stated; a couple of check boxes when you connect. I use it regularly and it doesn't get any easier! ..If only they could figure out a way to retrieve those papers printed over the internet! (but I guess thats why there is also options to make local drives / clipboard / printers / plug & play deveices / etc available in your RDP session)

HB007
HB007

I have the same error: I don't get the Certificate screen on my client but it connects anyway. Does anybody have any solution? Thank you in advance

rgilinsky
rgilinsky

As to clients, can I run remote desktop client on Home Basic? Does it ship with it? If yes and no, can I copy mstsc.exe from my XP Pro machine and run it on Vista Basic? (Same questiona re: Vista Home Premium.) My host is Windows 2003 Server Enterprise/SP2. It works now with Windows 2000 Pro and XP Pro clients.

ort
ort

I don't get the Certificate screen on my client but it connects anyway. Is there a way to force manuall certificate installation on the client prior to the connection (My host is a vista machine also) ? Or

randy.sigler
randy.sigler

You have to also remember that Vista is just Windows 2003 codebase with all the GUI *rap added. That is one reaons this is accessable on VISTA, it was already on Windows 2003 Server Code. Redmond could not get a brand new code base out the door for Vista ;-)

Editor's Picks