"Hello, my account keeps getting locked out"
Anyone who has admin for a Windows environment knows about accounts getting locked out. Typically when a user changes his or her password there are certain applications or processes that need to be manually updated with the new password. If you don't update these passwords, then each time they run, they will authentic your old password, which in turn will fail and register on the Active Directory as 1 failed password attempt.
If you think of a few more services and some mapped drives all trying to re-authentic at the same time, then you can easily be locked out if your Active Directory password policy is set to 5 failed passwords before account lockout. This problem while simple is often difficult to really track down where the problems are coming from.
I'm going to list the tools I use and a recent account lock out issue that I resolved. Please note I removed all of the details for security reasons.
What to look for
Before you make any changes, always gather the information of the user affected and what type of job role this user does. A data entry user will be much easier to resolve than someone who is an admin logging into many remote systems. Here's a listing of questions I usually ask.
1. When was the last time you changed your password?
2. Which account are you using?
3. Where do you login from, only this workstation or multiple computers?
4. Do you work from home on your personal computer?
5. Any special application or service you are using?
From here I can usually get an idea of the level for the user, let's say the user answered the following questions.
1. Two weeks ago, problem started today.
3. Only login from this laptop.
4. No, but I work from home on this laptop using VPN.
5. Google desktop search but nothing else.
Now let's check the user's desktop for any lock out messages. You can somewhat skip this part because the lock outs are from the Domain Controller but I like to just make sure I do this. As I checked the system, there is no indication of any security errors or lockouts. Strange, usually there's at least something to go on.
Ok, let's move to the Domain Controller. Checking the security logs I filter by the user name and again there is nothing listed for failed logins. I'm starting to wonder where is the lockouts coming from. The domain I am working on actually logs failed and successful from a group policy so with the amount of logs it could have been over written.
Time to install some tools
Before this I already installed the Windows Account Lockout and Management Toolset from Microsoft. I used the tools from our local Domain Controller and scanned the problem account with the LockOutStatus tool.
LockOutStatus shows a listing of the current status of the target account with details as where the lock outs are taking place and also what time they occur. Here's a screen shot of the tool and why each function is useful.
DC name - Shows which server the account is accessing
Site - Where the servers are located in the AD forest
User state - Is the account locked or unlocked
Bad pwd count - How many bad passwords were attempted
Last bad pwd - This is how long ago the last bad password was attempted
Pwd last set - When the password was last set
Lockout time - The time the accounted was locked out
What to look for?
So what I'm going to look for is this. First, I'm going to run the tool against the problem account, and the tool will automatically find all of the domain controllers. Here's what I look for in each value.
DC name - This is just the names of the DC, pair it up with the Site names if you are not sure where the servers are physically located.
Site - If you already know the DC name this is a bit non-important, unless you are not familiar with the DC's or do not know where a DC physical sits.
User state - This is handy to show the "wave" of lockouts getting updated as the account is locked out from it's primary DC. I alway focus on the DC that is first locked out, then if I can't find anything there move to the next closest DC.
Bad pwd count - This one is a bit tricky. In most cases you would see the number corresponding to your domain policy about bad password attempts. For example if your password count is 5 typically you will see "5" under this field. If you see 5 that would indicate that indeed there is an rogue application or process using an old password. Sometimes this does not mean that there were 5 attempts, but could mean that the DC saw 5 attempts. If you have some delay on your domain, 1 password attempt could register as 2 because of the delay between DC's.
Last bad pwd - This shows just when the account was locked out, for example if you know a user is working 8am to 5pm and the account lockout time shows 3am, then is must be a process running. If the user takes his or her laptop home and powers down at night, there might be a process running on a remote server.
Pwd last set - Of all of the values, I hold this second most important. Many times end users will change their passwords and that's when the problems will start. So I like to see the distance between when the password was set and when the password started to locking out. In most cases it's going to be very close, usually within a week from the password was last set. In some cases, it could be much longer because a service or application is not used every day.
Lockout time - Lockout time is when the DC reports the amount of failed password attempts to match the domain policy. It's a bit confusing and I'm not 100% but if an account is running on a service, and that service runs every 1 hour, it will register 1 lockout. But if the domain policy is set for 5 lockouts in one hour, this will not cause the account to be locked out.
So in this example, let's say we found the following values. We caught it just as the user's account was locked out so only one DC is showing locked.
DC name - DCSF
Site - San Francisco
User state - Locked
Bad pwd count - 5
Last bad pwd - 9:30AM
Pwd last set - 5/20/10 (two weeks ago)
Lockout time - 9:32AM
Ok, from this data I know the following. First, the account is contacting the DC in San Francisco, it's locking out from 5 attempts (5 is the setting in GPO), and it's locking out at 9:30AM. The user starts work at 9:15AM but he says the account is locked out before he can login to the workstation. This might be user error but let's dig further!
Checking the DC's security logs
Now we have a starting point, DCSF. From DCSF I filter out the logs and find this.
Here's the important information (removed real info).
Category: Account Management
Type: Success A
Event ID: 644
User Account Locked Out
Target Account Name: jsmith
Target Account ID: BayareaCo\jsmith
Caller Machine Name: sfauth
Caller User Name: DCSF$
Caller Domain: BayareaCo
Now this is important, the value for "Caller Machine Name" is pointing to "sfauth". I double checked and the user does not access any system besides his own. I checked the event logs of "sfauth" and found this.
Event ID: 539
Reason: Account locked out
User Name: jsmith
Logon Type: 3
Logon Process: CISCO
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGES_V1_0
Workstation Name: sfauth
We found the server, now what application is causing this
We now are focusing on the server sfauth since this last event log points to an process called "CISCO". Let's first see what is running on the sfauth. I'm going to use a handy tool from Sysinternals called TCPView. This shows me what processes and who they are talking to, as seen below.
So we found that the process is running but let's look at the process details. From TCPView, it looks like a process called "CSAuth.exe" is running on the server talking to DCSF. I'm going to assume that since it's talking to DCSF, it must be doing some kinda of authentication for remote access. I used another Sysinternals tool called Process Explorer.
We found the application, now what?
Ok, so we found that it's an Cisco application called Cisco Secure ACS, but what do we do now? I snoped around and found that the Cisco application had a few GUI tools and a nice logging report tool. I opened the repoting tool and viewed the failed logins.
Below is one of the failed logins reports for user "jsmith".
From here we have something to follow, a MAC address and an IP address. While the IP address can change, the MAC address is hard coded and unique for each network device. I first started by checking DHCP for where this IP address was located, it was not on the range of IP addresses. I used my favorite tool NMAP to do a port scan of the IP address listed in the failed event.
Below is the result, but there's something wrong with this picture, can you find it?
The MAC address listed here shows 00:13:19 while the logs shows 00:26:08, that's strange! Well if this device is a router then something else is accessing the network beyond what I could see. I did a quick search on a MAC address search tool and found this result.
Apple? We don't have any Apple devices here at work. Or do we? What is made by Apple and often times accessing the network AND would be accessing the network BEFORE the user would login to his computer?
Yes, the iPhone. The user has an iPhone which was accessing the WiFi network. In order to access the WiFi network you need to login with your AD account and this is held on the iPhone. Here's is the login screen for the WiFi access.
Solution and closure
Once I removed this from the user's phone access from the WiFi, the lockouts stopped. When I tried to recreate this problem I couldn't seem to get my iPhone to lockout but the end user did say he had a sync software running on his iPhone with his desktop. I'm glad I found the problem and resolved it. This by far was the most difficult account lockout I have battled.
I hope this will be helpful for you!