SolutionBase: Diagnosing Kerberos health issues, part 1

You can greatly increase security on your Windows 2003-based network by using Kerberos. When you do, however, sometimes it's difficult to figure out what's going wrong with it when it doesn't work. In this article, Brien Posey shows you how you can use Windows Server 2003 tools to diagnose Kerberos problems.

This article is also available as a TechRepublic download.

Most Windows administrators know that Windows server uses Kerberos for authentication. The problem is though, how can you be sure that Kerberos is working properly? After all, there aren't any Kerberos diagnostic tools on the Administrative Tools menu. Likewise, you'll never see a pop-up message telling you that Kerberos is malfunctioning, and even Kerberos related event log entries tend to be scarce. So how can you tell the Kerberos is even running? In this article, I will answer that question by showing you some diagnostic tools that you can use to make sure that Kerberos is working properly.

Before I begin

Before I get started, I just want to mention that most of the tools that I will be discussing are included as a part of the Windows Server 2003 Resource Kit. If you do not have a copy of the Windows Server 2003 Resource Kit, then you can download the resource kit tools for free Microsoft's Web site.

For the purposes of this article, I'm assuming that you have already downloaded the resource kit tools, and installed them to the default directory.

A crash course in Kerberos

Before start showing you the various diagnostic techniques, I want to take a few minutes and briefly discuss how Kerberos works. After all, it can be tough to perform an accurate diagnostic if you don't even understand what it is that you're diagnosing.

The name Kerberos actually came from Greek mythology. Kerberos was the name of a three headed dog whose job it was to guard the entrance to Hades (some text uses the name Cerberus instead). Using the name of the three headed attack dog is actually kind of appropriate when you stop and think about what it is the Kerberos does. Kerberos is an authentication protocol that is designed to protect authentication credentials from network sniffers.

As I mentioned before, Windows 2000, 2003, and XP are all Kerberos enabled. When a Kerberos enabled client needs to request a network service, such as a logon, from a Kerberos enabled server it begins the process by contacting an authentication server. In a Windows environment, this authentication server is simply a domain controller. The authentication server issues the client a Kerberos ticket and an encryption key (also known as a session key).

The Kerberos ticket that is initially issued by the authentication server contains a copy of the encryption key and a randomly generated identity number. When the client receives this ticket, it stores the ticket and its ticket cache. When the client meets to access the network service, it transmits a copy of the ticket to a ticket granting server (the ticket granting server and the authentication server are almost always the same machine). The ticket granting server then sends the client another ticket. This ticket identifies the client a service that it is requesting. The client then gives this ticket to the network service that it is trying to access and is granted access.

In case you're wondering, the client does not have to go through this process each time that it needs to access a resource. Once the ticket granting server has given the client a ticket for a particular resource, the client can use the ticket repetitively until either the ticket expires or the user logs off the client machine.

Another thing worth mentioning is that when the client uses a ticket to gain access to a resource, the client also sends the resource server and authenticator message. This authenticator message includes a time stamp and is encrypted with the session key. Its purpose is to prove that the ticket is legitimate.

Why run Kerberos diagnostics?

Now that I have given you a brief overview of how Kerberos authentication works, I want to talk about Kerberos diagnostics. If your system isn't having any obvious problems, then it may seem pointless to the running Kerberos diagnostic utilities. However, I tend to believe that running a Kerberos diagnostic once in awhile is worthwhile, even if there are no obvious problems. There are several reasons for this.

For starters, you need to be able to verify that Kerberos is even running. Yes, Kerberos is the preferred authentication protocol for Windows 2000, 2003, and XP. However, just because Kerberos is the preferred authentication protocol it does not mean that it is the only authentication protocol. All of the operating systems that I just mentioned also support the older and less secure NTLM authentication. This is why it's important to make sure that Kerberos is actually being used.

It's also important to understand what Kerberos related events actually mean. Some Kerberos errors actually reflect normal system functions. Other types of errors though can point to a Kerberos malfunction, or can be a symptom of an attack against your system. You need to be able to tell the difference. If you do detect that Kerberos malfunction, that malfunction may not be directly related to Kerberos itself, but rather to an underlying dependency such as DNS or the Active Directory in general.

Network Monitor

One of the primary tools that we will use to verify Kerberos functionality is Network Monitor. Network Monitor is included with Windows Server 2003, but it's not installed by default. To install Network Monitor, select the Add or Remove Programs command from the Control Panel. When the Add or can't remove Programs Control Panel applet opens, click the Add/Remove Windows Components button. After a brief delay, the Windows Components Wizard will open.

Scroll through the list of available components until you find the Management and Monitoring Tools option. Select this option, and click the Details button. You will now see a list of the available management and monitoring tools. Select the Network Monitoring Tools check box and click OK. Now, click Next and Windows will begin copying the necessary files. There is a chance that you may be prompted for your Windows installation CD during this process. When the file copy process completes, click the Finish button to close the wizard.

It is worth noting, but the version of Network Monitor that comes with Windows Server 2003 is limited. This version of Network Monitor will only display communications are occurring directly between the server that Network Monitor is installed on, and other computers that are accessible through a designated network interface. What this means is that depending on how your network is configured, your copy of Network Monitor may not display all of the information that I'm about to show you. However, there is a full featured version of Network Monitor that comes with SMS Server. I will be using the SMS Server version of Network Monitor throughout the duration of this article.

Kerberos at boot-up

If you ever worked through the process of joining a computer to a domain, then you probably know that every computer within a domain has a domain account of its own that a separate from the user account. When a computer is booted, this computer account is used to authenticate the computer into the domain. Again, this is completely separate from the user log in. Even so, Kerberos is used during the computer's authentication process. In this section, I'm going to show you how you can use Network Monitor to watch the authentication process and verify that Kerberos is being used.

Begin the process by shutting down a PC is a member of the same domain as the one that you're running Network Monitor on. After doing so, open Network Monitor. You can find it on the servers Administrative Tools menu.

Upon loading Network Monitor, you'll see a message asking you to specify the network on which you want to capture data. Click OK, and you'll be presented with the list of all of the server's network interfaces. Select the network interface through which the workstation that you powered off earlier will communicate with the server, and click OK. You will now see the main Network Monitor screen, but Network Monitor will not be tracking any data yet.

When you're ready to analyze the boot process, select the Start command from Network Monitor's Capture menu. When you do, Network Monitor will immediately begin capturing packets. Now, reboot the workstation that you turned off earlier. When the workstation reaches the login prompt, go back to the server and select the Stop command from the Capture menu.

The Network Monitor buffer should now contain information related to the workstations authentication onto the domain. You must now sift through that information to make sure that Kerberos is being used, and that it is functioning correctly.

The first thing that you need to do is to put the captured data into a more usable format. If you look at the Network Monitor taskbar, you'll see an icon that looks like a pair of eyeglasses. Click this icon, and you will see an expanded view of the captured data. Now, select the Zoom Pane command from the Window menu. When you do, you will see a screen similar to the one shown in Figure A. You can adjust the size of each pane to make viewing data easier.

Figure A

This is what the captured data looks like.

Now let's go through the captured data and see what the authentication process looks like. When a domain member boots, it must authenticate itself to the domain controller the first step in this process is to perform a series of DNS queries that register at the computer's name with the DNS server, and locate a domain controller for the domain. You can see some of these queries shown in Figure B.

Figure B

DNS queries are used to register the computer and to locate a domain controller.

After the necessary DNS queries have been performed, the server makes an LDAP query against a domain controller looking for the Netlogon service. If you look at Figure C, you can see that frame number 125 is performing the LDAP query for the Netlogon service. You can see the reference to Netlogon at the very bottom of the middle frame. As you look at this figure, you might also take a look at frame number 126. This frame contains the domain controller response to the LDAP query.

Figure C

The computer performs a LDAP query to find the Netlogon service.

 After the Netlogon service has been located, the computer will send the UDP query over port 88. This is the first instance of Kerberos being used. This UDP query is the computer's request for a ticket granting ticket. In this query, the computer is using its computer password as a key in a cryptographic hash of a timestamp. In addition to this hash, the query also includes a plane text copy of the timestamp. You can see what this query looks like through Network Monitor by looking at Figure D.

Figure D

This is the Kerberos request for a ticket granting ticket.

When the domain controller receives the query, it compares the unencrypted time stamp against the domain controller's own clock. If there is more than a five minute difference between the two times, then the query is rejected (NTLM might be used as an alternative though).

Assuming that the timestamps are at least reasonably close, the domain controller uses its copy of the computer account's password to create a cryptographic hash of the unencrypted time stamp. The domain controller then compares the hash that it created against the hash that was sent to it by the computer that made the query. If the two hashes match, then the computer's identity is authenticated.

When the domain controller validates the computer's identity, it transmits a ticket getting ticket back to the computer within a UDP packet. The computer can then use the ticket that has been returned to it in order to request services from other servers on the network. In fact, if you look at Figure D, you will see that there is a whole series of Kerberos transactions taking place. Many of these transactions are requests for service for various servers.

If we scroll down through the Network Monitor's captured packets, there is eventually another series of Kerberos transactions taking place, as shown in Figure E. This is the user logon related traffic. The entire process happens all over again when the user logs in. The timestamp is encrypted using the user's password as the key and is sent to the domain controller along with an unencrypted timestamp. The domain controller makes a hash using the unencrypted timestamp and its own copy of the user's password and authenticates the user's password. After the password is authenticated, the domain controller issues the user a ticket getting ticket.

Figure E

Another set of Kerberos transactions occur when the user logs on.

Right now you might be wondering how you can tell these Kerberos transactions apart. After all, Kerberos traffic is encrypted, so you can't exactly read all of the details through Network Monitor.

The easiest way to figure out what is going on is to use the time stamps. Although it is not fully expanded in my screen captures, Network Monitor contains a Time column that shows what time each packet was sent. If you make note of the time of a particular transaction, you can look in your domain controller's audit logs to see what was happening at that time. If you have multiple domain controllers though, then any one of them could have been the one to authenticate the user or computer.

For example, if you look at Figure F, you can see that this particular packet was sent at 8:28:26 AM. If you look further down the screen, you can also see the source and destination IP addresses. The source address in this case just happens to be one of my domain controllers. I can therefore go to my domain controller and check the audit logs for 8:28:26. As you can see in Figure G, the transaction corresponds to an Authentication ticket request for a user named Taz.

Figure F

If you examine a packet in detail, you can see its timestamp, and its source and destination address.

Figure G

You can use your domain controller's audit logs to verify what Network Monitor is showing you.