Understand Windows authentication to improve security

A thorough understanding of Windows' authentication methods will enable you to troubleshoot problems and improve network security. Here's a look at what you need to know.

If you're having problems logging in to a diverse and properly secured network, the solution may be just a matter of allowing the proper ports and protocols between the devices to authenticate. Before jumping into the ports and protocols, however, it's important to understand the sequence of events that begin user authentication.

The WinLogon process
WinLogon's first phase is [Ctrl][Alt][Delete], Windows' default Security Attention Sequence (SAS). This sequence signals to the operating system that someone is trying to log on.

When SAS is initiated, all user mode applications pause until the security operation is completed or cancelled. This suspension of user mode applications is a significant security feature. Keystroke loggers or Trojan viruses are disabled and prevented from recording keystrokes as users input their passwords.

The WinLogon process is a part of the Local Security Authority (LSA) for the Windows operating system logon procedure. To complete this procedure, the OS authenticates the user's credentials with a logon server and, depending on the type of authentication, the logon could fail if the proper ports and protocols between the client and the server aren't open.

NT LAN Manager (NTLM) is the default authentication scheme used by the WinLogon process; it uses three ports between the client and domain controller (DC):
  • UDP 137 - UDP 137 (NetBIOS Name)
  • UDP 138 - UDP 138 (NetBIOS Netlogon and Browsing)
  • 1024-65535/TCP - TCP 139 (NetBIOS Session)

Logon authentications will succeed with these ports open between your clients and their domain controllers.

Windows' default authentication equals poor security
By default, the authentication scheme on both Windows NT and Windows 2000 machines is set to LAN Manager (LM), which transmits and stores each user password hash using an extremely weak security algorithm. This makes it easy for hackers to crack passwords once this weak password hash is captured.

Microsoft has upgraded its proprietary authentication scheme four times. The current authentication standard for communications between NT clients and NT/Win2K servers is now NTLMv2. However, if you haven't changed the LMCompatabilityLevel variable under the following registry key on both NT/Win2K clients and servers, by default you're still using the LM scheme—which greatly decreases the security of your entire network:

Make sure that you're set to use only NTLMv2 and that the Reg_Dword is set to at least Level 3. This forces the clients to send NTLMv2 authentication only. (For more information on this change, read Microsoft Knowledge Base article 147706.)

Author's note
Editing the system registry is risky. Before making any registry edits, be sure to back up the registry so you can restore it if something goes wrong.

After you make this change, you'll still need to force the system to remove the LM hash. Run Regedt32.exe and go back to the same registry key. On the Edit menu, click Add Key, add the key NoLMHash, and leave the class field empty. (For more information on this registry change, read Microsoft article 299656.)

This new registry key will force both NT and Win2K to remove the LM hash, which will reduce your vulnerability to password crackers. However, any registry changes you make won't take place until the user changes his or her password and a new hash is created.

Now that you've gotten rid of the LM hash and your network is using NTLMv2 for client authentication, the next step for securing authentication traffic on Windows-based networks is upgrading your clients and servers to take advantage of Kerberos, the latest Windows authentication scheme.

What about Kerberos for NT?
There is no such thing as "Kerberos for NT." If you're still running NT clients, you can install the Active Directory (AD) client and make them Active Directory-aware, but not Kerberos-enabled. The AD client installs the Active Directory Service Interfaces (ADSI), provides site awareness to NT computers so they can find the nearest domain controller, and enables NT to use Win2K's Dfs and AD Windows Address Book. This doesn't change the method NT computers use for authentication.

NT machines will use only NTLM to authenticate, regardless of whether they're communicating with NT or Win2K servers. To allow your clients and servers to communicate securely, make sure the proper ports are open and verify that your clients and servers are set to use NTLMv2.

This article originally appeared in TechRepublic's Security Solutions e-newsletter.


Editor's Picks