Developing ways to reduce and resolve Active Directory account lockouts has been one of my quests lately. I wrote recently about how to reduce account lockouts and password resets and how to troubleshoot account lockouts via Group Policy. Both were based upon real-world events or ideas from day-to-day operations.
Bad events often seem to occur in threes, so as it turned out I faced a more complicated issue along the same theme. The issue and how I resolved it is worthy of commentary since it illustrates the complexity of troubleshooting and how to proceed in a sane and orderly fashion.
It all started after a routine password change of my Active Directory account. Everything seemed fine for a day or so, then when I logged into the VPN from home and tried to connect to my work PC via Remote Desktop I got this error:
Microsoft isn’t always great about telling you exactly why something isn’t working. In this case the password was correct, but the account was locked. Fortunately I’ve come across this issue before, so I didn’t waste time resetting my password since that’s a red herring here.
This article is going to give an in-depth explanation of how to resolve issues like this, but first, here’s the short version:
- In a nutshell, I found that my account was locking out via failed authentication attempts to our Exchange server.
- I used a tool called ExMon to determine what network address the connection was coming from and determined these were originating from outside the company via a mobile device.
- I then ran an Exchange Powershell command called Get-ActiveSyncDeviceStatistics to show ActiveSync devices associated with my account and discovered a forgotten tablet in the server room was trying to authenticate to my mailbox using a months-old password.
And now, here’s the whole story:
After my initial account lockout, I logged in with another domain administrator account and unlocked it, but so began started a troubling crusade to stop my account from locking again and again. Sporadic lockouts began occurring, varying in time from every 15 minutes to once per day. Connections are permitted both from internal and external systems, adding to the complexity.
As a system administrator, I’m often logged into many workstations and servers at once, so I assumed I was logged in somewhere with my prior password (but I also wondered why the lockouts were sporadic when they should have been consistent). You can’t go rebooting systems left and right, of course, trying to play whack-a-mole to solve an issue like this. Instead I followed the steps in my “How to troubleshoot account lockouts via Group Policy” article and determined the IP address from which the authentication failure occurred:
In this case the IP address was that of our company Exchange server. Problem solved; I must be logged in there, I thought. I thought wrong.
Eliminate what you can
It’s easy to conclude that if your Active Directory account is locking out via Exchange then you must have Outlook running on a workstation somewhere and logon failures are producing the problem. What troubled me was that I found no failed authentications from workstations, which do indeed authenticate users against Active Directory. Had my theory been correct, I should have found 4771 errors from multiple PCs (which is why it’s important to assess available information and rule out as many irrelevant notions or ideas as possible).
My company PC kept popping up an Outlook password prompt which then failed to accept my new password, so I thought perhaps the Outlook profile itself had an issue, such as by caching the old password somehow. I removed and recreated the profile to no avail; the account lockouts persisted. This became especially frustrating and problematic during a couple of urgent situations where I needed to log in immediately, or when I couldn’t get business email on my phone. In both cases I had to log in with an alternate account then unlock mine.
I considered setting back to old password or even creating a new account to see if that alleviated the issue. I didn’t want to go backwards, however, but as an experiment I tried setting my account to use the old password – and the lockouts continued. I then reverted my password to the new one since I didn’t want to introduce other weirdness.
I wised up and shut down Outlook on my PC then monitored the domain controller for account lockouts. When they kept occurring I knew I could rule out my Outlook client, but just to make certain my PC had nothing to do with the problem I took it offline and found the problem persisted. I knew my system had nothing to do with the situation. I left it offline for the time being.
Use the right tools
It became apparent the way to solve the issue was to figure out what was connecting to the Exchange server to access my account. The Exchange Server User Monitoring Tool (ExMon) can come in handy by displaying the source IP addresses related to mailbox access. It only monitors MAPI connections (such as from mail clients), not the SMTP, POP or IMAP protocols, but I didn’t assume those were related to the problem.
There are a couple of available versions of ExMon; ExMon for Exchange 2000-2010 and
ExMon for Exchange 2013 and 2016. It’s worth noting that Microsoft has documented how to using auditing to see the IP addresses of clients connecting to Office 365 accounts so this technique may help as well if you’re using cloud-based Exchange.
I got the correct version loaded up then fired it up:
The tool monitors for client connections and refreshes by default every minute, but you can refresh manually by clicking File then Refresh Now, or by using the refresh icon in the toolbar (under Help). This can be necessary if you don’t see the target mailbox listed right away; the statistics change quite rapidly if you have many clients talking to Exchange.
It was time to start digging. And the answer became apparent right away. I saw my own mailbox (smatteso) was being accessed by an IP address which corresponded to the internal IP address of our company firewall.
In other words, the connection was coming from the outside!
Pinpoint the problem
A good security practice is to permit only authorized devices to connect to company email from outside, utilizing passwords and/or certificates, and to “quarantine” unknown devices until IT staff can confirm their legitimacy and approve their access. Once these can contact your mailbox you can then see which devices have done so.
Exchange offers a slew of handy Powershell scripts which can help you to quickly locate the source of various problems. While many of them duplicate available functionality in the GUI (such as via the Exchange Management Console), the command line offers more power and versatility, not to mention better speed.
In my case I ran the “Get-ActiveSyncDeviceStatistics -Mailbox smatteso | ft DeviceType, DeviceUserAgent, LastSuccesSync” command to take a look at which devices were connecting to my mailbox over ActiveSync, which is the communication/synchronization protocol used by Exchange. Piping the comment to “ft” formats the output nicely and specifying only the categories I needed to know – the device type, what agent was running on the device, and when it last synchronized.
Right off the bat I had my answer (note: I intentionally enlarged the date of that bottom entry to emphasize its significance.)
Amidst a slew of old devices (former phones of mine) I could see that a Samsung-SM-G935P, also known as a Samsung Galaxy S7 Edge, and a Samsung-SM-T217S device had attempted to access my mailbox; the first just minutes before and the second last March.
I recognized the Edge; that’s my phone. The other one is a Samsung Galaxy Tab which produced a facepalm moment for me, since that’s a test tablet which was in our data center.
Everything suddenly made perfect sense. I had set up my work email account on that tablet a while back then left it in the server room, which has sporadic wi-fi connectivity due to varying environmental conditions and the location of the tablet. The tablet had been connecting on occasion to my mailbox to try to authenticate and kept failing. It had an old stored password from way back, which explains why setting the password back to the prior one didn’t help. Lastly, this issue hadn’t popped up before since the laptop ran out of power in mid-March and had recently plugged it in to charge up, thereby creating the password problem.
So in summary, these were the important clues uncovered:
- The lockouts occurring only via the Exchange server pointed against a workstation issue.
- Taking my workstation offline confirmed the above clue.
- The sporadic lockouts signified an unreliable or inconsistent connection.
- The old password still being rejected indicated a completely erroneous or outdated credential.
- Writing down the clues and focusing on how they connect can help cut down on problem resolution times.
- Discussing them with other team members to brainstorm can be beneficial as well.
This was a good learning experience on how to keep track of and remain aware of endpoint devices, even those in a secured location. I removed my email account from the tablet, removed all those outdated devices associated with my mailbox then documented the whole process for my company so that future instances of this nature can be readily addressed. Always leave a trail behind you for the next would-be sleuth traveling the same path!
Also see:
- How to create stronger passwords by using data-driven feedback (TechRepublic)
- How to make your employees care about cybersecurity: 10 tips (TechRepublic)
- The dumbest passwords people still use (ZDNet)
- Ethical Password Hacking and Security (TechRepublic Academy)
