SolutionBase: Troubleshooting SSL encryption problems

SSL helps make your Web sites more secure -- when it works. Here are some tricks you can use to troubleshoot typical SSL encryption problems.

Although not officially regulated by IEEE or any other organization, Secure Sockets Layer (SSL) has been the primary mechanism for securing Internet-based communications for many years now. Unfortunately, SSL can be a little tricky to configure, and when things go wrong, you may not always get an error message. Instead, clients might simply not be able to connect to the site. To help get rid of the headaches associated with troubleshooting SSL problems, Microsoft has created the SSL Diagnostic Utility.

Acquiring the SSL Diagnostic Utility

You can download the SSL Diagnostic Utility directly from Microsoft's Web site. The download consists of a 2.06-MB setup.exe file. To install the utility, copy it to an empty folder on your IIS server and double-click the setup.exe file. Windows will launch the installation wizard. Click Next to bypass the wizardï¿?s welcome screen. After accepting the end-user license agreement, click Next and then enter your name and organization. Click Next once again and enter the installation path. Finally, click Next, followed by Install, to begin copying the necessary files. When the copy process finishes, click Finish to complete the installation.

A crash course in SSL

Before I get too far into my discussion, I want to give you a crash course in SSL encryption so you'll understand the concepts addressed later. If you already know how SSL works, feel free to move on to the next section.

There are several ways SSL can be configured and used. For the purposes of this article, I'll assume that SSL is used in an environment in which an administrator needs to secure communications, but has no control over the clients. A classic example of this is e-commerce. For example, you can easily set up an e-commerce Web site, but you have no way of controlling how visitors to the site have their PCs configured. Since secure communications are important during the transaction, you'd rely on SSL encryption.

SSL encryption works because a digital certificate is assigned to the server. This certificate serves two main purposes. It guarantees site visitors that you are who you say you are. The certificate is usually provided by a trusted third-party certificate authority who goes to great lengths to verify a person or companyï¿?s identity before assigning them a certificate.

The certificate also provides the basis for encryption. When a certificate is assigned to the server, the server is given a public key and a corresponding private key. The private key is a closely guarded secret, but the server will share the public key with anyone who asks for it.

When a client needs to establish an SSL connection, the client begins by requesting the serverï¿?s public key. The server then sends the public key to the client. The client is able to use this key to create an encrypted message that can be decrypted only by using the serverï¿?s private key. This allows the client to send secure, encrypted messages to the server. The problem is that the server must also be able to send encrypted messages back to the client, and the client doesnï¿?t usually have its own public/private key combination to facilitate encrypted server-to-client communications.

To get around this problem, SSL makes use of a session keyï¿?a shared key thatï¿?s dynamically generated by the client. Since the client can securely communicate with the server, the client generates a session key, encrypts it with the serverï¿?s public key, and then sends the session key to the server. When the server receives the session key, it is decrypted and used for all remaining encrypted messages flowing between the two machines. Since both the client and the server now have a copy of the session key, either machine can send encrypted data to the other.

Using the utility

To launch the SSL Diagnostic Utility, simply double-click the ssldiag.exe icon located in the folder in which you installed the utility. The utility can be a little intimidating at first. When I initially used it on my test server, the utility began spewing loads of information. When I took a second look, though, I realized that the reason for the mass flood of information was because my test server was configured to host so many different Web sites.

If you look at Figure A, you'll see that unless a site has a certificate assigned to it, the utility presents you with only three or four lines of information per site. The first line looks something like [W3SVC/5]. You can ignore the W3SVC portion of this line, because it simply indicates that the site is running on IIS and has the W3C extended logging format enabled. The number that comes after the W3SVC/ indicates the site number. Therefore, if a line reads [W3SVC/5], that line and the following lines reference the fifth site being hosted by the server.

Figure A

The SSL Diagnostic Utility reports on each site being hosted by the server.

Next comes the ServerComment line, which tells you the name of the Web site the section pertains to. For example, in Figure A, the site W3SVC/5 is related to a site named ProjectX.

The ServerAutoStart line tells you whether the site is designed to automatically start when the server starts. Not every site will display this line. Finally, the ServerState line tells you whether the Web site is running, as you can see in Figure A.

The utility will display the above information whether a site is configured to use SSL or not. If a site is configured to use SSL, though, you'll also see the information shown in Figure B. This information pertains to the certificate assigned to the server. It's basically the same information you'd see if you clicked the View Certificate button within the Internet Services Manager.

Figure B

The utility will display information about the serverï¿?s certificate if one has been assigned.

Although the utility is displaying basic information about the certificate in Figure B, there's one thing thatï¿?s really worth paying attention to. If you look at the last line of the [W3SVC/9] section, you'll notice an exclamation point next to the #SSL Port line and an error message stating that the SecureBinding property is not set. The utility has flagged the reason why SSL is not working properly on the Web site.

If you donï¿?t know what this error message means, itï¿?s not a problem. If you select the error message, another message appearing near the bottom of the user interface will guide you through the solution to the problem, as shown in Figure C. In this case, the utility tells me that I forgot to tell IIS what port number I wanted to use for SSL communications.

Figure C

The utility displays the problem and provides you with a step-by-step solution.

After performing the suggested fix, go back to the utility and click the Refresh button. The utility will reanalyze the Web sites and display updated information. As you can see in Figure D, the error message has been replaced by a message stating that secure bindings have been set to

Figure D

After performing the suggested repair steps, click the Refresh button to see if the problem goes away.

Certificate problems

In this case, I purposely misconfigured an IIS server so that you could see how the utility points out a problem and guides you through fixing it. Sometimes, however, the utility might not point out an error that's due to an SSL malfunction. When this happens, there's a good chance that the problem could be with your certificate rather than with the way IIS is configured.

Fortunately, the utility provides you with an easy way to tell whether the serverï¿?s certificate is the cause of the problem. The utility can create a self-signed certificate that is guaranteed to be valid and can then temporarily replace your existing certificate with the known good certificate. If the problem disappears, you know that your certificate was bad. If the problem persists, then it probably isnï¿?t related to the certificate itself (although it's possible to have a bad certificate and some other problem too).

To try using a known good certificate, right-click the [W3SVC/#] line in the utility and select the Create New Cert command from the resulting shortcut menu. The utility will automatically refresh its display, and you can verify that your Web site is now using the alternate certificate.

Once the alternate certificate is in place, go to a client machine and try to access the secure Web site through the HTTPS protocol. You'll see a warning similar to the one shown in Figure E. This warning message tells you that the certificate wasnï¿?t issued by a trusted certificate authority and that the name on the certificate doesnï¿?t match the name of the site. This is perfectly normal since the certificate is designed for testing purposes and not as a permanent replacement for the siteï¿?s real certificate.

Figure E

This warning is normal when using a test certificate.

After you complete your testing with the known good certificate, you'll probably want to restore your original certificate. The nice part is that you donï¿?t have to actually restore the certificate because it wasnï¿?t really deleted. All you have to do is reconfigure which certificate is associated with the site. The easiest way is to right-click the [W3SVC/#] line in the utility and select the Restore Cert command from the resulting shortcut menu.

A closer look

I hope the information I've given you so far has helped you solve any SSL problems. If SSL encryption is still not working correctly, however, the SSL Diagnostic Utility has another feature you can use to try to troubleshoot the problem: the SSL Handshake Simulation.

The SSL handshake refers to the process of sending the various encryption keys back and forth, as described in the earlier section "A crash course in SSL." The SSL Handshake Simulation allows you to launch an SSL-encrypted session with the server in the same way that a client would from a Web browser. But instead of simply displaying the resulting Web page, the SSL Diagnostic Utility will show detailed information about the handshake process, including any errors that might occur.

To simulate the handshake process, right-click the [W3SVC/#] line and select the Simulate SSL Handshake command from the resulting shortcut menu. The utility will open a new window and walk you through the handshake process.

For demonstration purposes, I've removed the SSL port number from the IIS configuration (as was the case in Figure B). In Figure F, you'll see that the handshake process terminated immediately because of this error. Once again, if you select the error, the utility will guide you through the repair process.

Figure F

The handshake simulator will provide you with details of the handshake process.

In case you're wondering, a successful handshake looks something like the one shown in Figure G. Notice that once the handshake is complete, the simulator begins to display the siteï¿?s HTML code.

Figure G

When a handshake is successful, the simulator begins displaying the siteï¿?s HTML code.

Client certificates

You can use the SSL Diagnostic Utility to diagnose problems with client certificates as well as with server certificates. Client certificates are analyzed through a feature called the Client Certificate Monitor.

This feature can monitor the usage of client certificates in real time by attaching to the actual process where encryption and decryption occur. As the encryption or decryption action takes place on the server, the Client Certificate Monitor displays the certificates being used by the clients and the information contained within those certificates.

More important, though, the monitor shows the result of each SSL connection attempt. This means that you'll be able to tell which certificates were valid or invalid. If a certificate is deemed to be invalid, the reasons for the invalidity are displayed. Typical reasons why a certificate is invalid are that it has expired, is not yet valid, or has been revoked.

Most organizations will probably never need to run the Client Certificate Monitor because they rely purely on server certificates. Client certificates are used only on Web applications in which a clientï¿?s identity must be proven beyond a shadow of a doubt, and the administrator has direct control over or access to the clientï¿?s machines.

If you find yourself in need of running the Client Certificate Monitor, you must remember that because it interacts directly with the encryption/decryption processes, it places a significant burden on the server. I therefore recommend using the monitor sparingly. Microsoft also recommends rebooting your server after you finish using the monitor.