Friday, June 04, 2010

Mysterious Windows lockout

"Hello, my account keeps getting locked out"

Anyone who has admin for a Windows environment knows about accounts getting locked out. Typically when a user changes his or her password there are certain applications or processes that need to be manually updated with the new password. If you don't update these passwords, then each time they run, they will authentic your old password, which in turn will fail and register on the Active Directory as 1 failed password attempt.

If you think of a few more services and some mapped drives all trying to re-authentic at the same time, then you can easily be locked out if your Active Directory password policy is set to 5 failed passwords before account lockout. This problem while simple is often difficult to really track down where the problems are coming from.

I'm going to list the tools I use and a recent account lock out issue that I resolved. Please note I removed all of the details for security reasons.

What to look for

Before you make any changes, always gather the information of the user affected and what type of job role this user does. A data entry user will be much easier to resolve than someone who is an admin logging into many remote systems. Here's a listing of questions I usually ask.

1. When was the last time you changed your password?
2. Which account are you using?
3. Where do you login from, only this workstation or multiple computers?
4. Do you work from home on your personal computer?
5. Any special application or service you are using?

From here I can usually get an idea of the level for the user, let's say the user answered the following questions.

1. Two weeks ago, problem started today.
2. Jsmith
3. Only login from this laptop.
4. No, but I work from home on this laptop using VPN.
5. Google desktop search but nothing else.

Now let's check the user's desktop for any lock out messages. You can somewhat skip this part because the lock outs are from the Domain Controller but I like to just make sure I do this. As I checked the system, there is no indication of any security errors or lockouts. Strange, usually there's at least something to go on.

Ok, let's move to the Domain Controller. Checking the security logs I filter by the user name and again there is nothing listed for failed logins. I'm starting to wonder where is the lockouts coming from. The domain I am working on actually logs failed and successful from a group policy so with the amount of logs it could have been over written.

Time to install some tools

Before this I already installed the Windows Account Lockout and Management Toolset from Microsoft. I used the tools from our local Domain Controller and scanned the problem account with the LockOutStatus tool.

LockOutStatus shows a listing of the current status of the target account with details as where the lock outs are taking place and also what time they occur. Here's a screen shot of the tool and why each function is useful.


DC name - Shows which server the account is accessing
Site - Where the servers are located in the AD forest
User state - Is the account locked or unlocked
Bad pwd count - How many bad passwords were attempted
Last bad pwd - This is how long ago the last bad password was attempted
Pwd last set - When the password was last set
Lockout time - The time the accounted was locked out

What to look for?

So what I'm going to look for is this. First, I'm going to run the tool against the problem account, and the tool will automatically find all of the domain controllers. Here's what I look for in each value.

DC name - This is just the names of the DC, pair it up with the Site names if you are not sure where the servers are physically located.

Site - If you already know the DC name this is a bit non-important, unless you are not familiar with the DC's or do not know where a DC physical sits.

User state - This is handy to show the "wave" of lockouts getting updated as the account is locked out from it's primary DC. I alway focus on the DC that is first locked out, then if I can't find anything there move to the next closest DC.

Bad pwd count - This one is a bit tricky. In most cases you would see the number corresponding to your domain policy about bad password attempts. For example if your password count is 5 typically you will see "5" under this field. If you see 5 that would indicate that indeed there is an rogue application or process using an old password. Sometimes this does not mean that there were 5 attempts, but could mean that the DC saw 5 attempts. If you have some delay on your domain, 1 password attempt could register as 2 because of the delay between DC's.

Last bad pwd - This shows just when the account was locked out, for example if you know a user is working 8am to 5pm and the account lockout time shows 3am, then is must be a process running. If the user takes his or her laptop home and powers down at night, there might be a process running on a remote server.

Pwd last set - Of all of the values, I hold this second most important. Many times end users will change their passwords and that's when the problems will start. So I like to see the distance between when the password was set and when the password started to locking out. In most cases it's going to be very close, usually within a week from the password was last set. In some cases, it could be much longer because a service or application is not used every day.

Lockout time - Lockout time is when the DC reports the amount of failed password attempts to match the domain policy. It's a bit confusing and I'm not 100% but if an account is running on a service, and that service runs every 1 hour, it will register 1 lockout. But if the domain policy is set for 5 lockouts in one hour, this will not cause the account to be locked out.

So in this example, let's say we found the following values. We caught it just as the user's account was locked out so only one DC is showing locked.

DC name - DCSF
Site - San Francisco
User state - Locked
Bad pwd count - 5
Last bad pwd - 9:30AM
Pwd last set - 5/20/10 (two weeks ago)
Lockout time - 9:32AM

Ok, from this data I know the following. First, the account is contacting the DC in San Francisco, it's locking out from 5 attempts (5 is the setting in GPO), and it's locking out at 9:30AM. The user starts work at 9:15AM but he says the account is locked out before he can login to the workstation. This might be user error but let's dig further!

Checking the DC's security logs

Now we have a starting point, DCSF. From DCSF I filter out the logs and find this.

Lockout event

Here's the important information (removed real info).

Source: Security
Category: Account Management
Type: Success A
Event ID: 644
Computer: DCSF
User Account Locked Out
Target Account Name: jsmith
Target Account ID: BayareaCo\jsmith
Caller Machine Name: sfauth
Caller User Name: DCSF$
Caller Domain: BayareaCo

Now this is important, the value for "Caller Machine Name" is pointing to "sfauth". I double checked and the user does not access any system besides his own. I checked the event logs of "sfauth" and found this.

Source: Security
Category: Logon/Logoff
Event ID: 539
Computer: sfauth
Reason: Account locked out
User Name: jsmith
Domain: BayareaCo
Logon Type: 3
Logon Process: CISCO
Workstation Name: sfauth

We found the server, now what application is causing this

We now are focusing on the server sfauth since this last event log points to an process called "CISCO". Let's first see what is running on the sfauth. I'm going to use a handy tool from Sysinternals called TCPView. This shows me what processes and who they are talking to, as seen below.

TCP View

So we found that the process is running but let's look at the process details. From TCPView, it looks like a process called "CSAuth.exe" is running on the server talking to DCSF. I'm going to assume that since it's talking to DCSF, it must be doing some kinda of authentication for remote access. I used another Sysinternals tool called Process Explorer.


We found the application, now what?

Ok, so we found that it's an Cisco application called Cisco Secure ACS, but what do we do now? I snoped around and found that the Cisco application had a few GUI tools and a nice logging report tool. I opened the repoting tool and viewed the failed logins.

Below is one of the failed logins reports for user "jsmith".

Cisco Login

From here we have something to follow, a MAC address and an IP address. While the IP address can change, the MAC address is hard coded and unique for each network device. I first started by checking DHCP for where this IP address was located, it was not on the range of IP addresses. I used my favorite tool NMAP to do a port scan of the IP address listed in the failed event.

Below is the result, but there's something wrong with this picture, can you find it?


The MAC address listed here shows 00:13:19 while the logs shows 00:26:08, that's strange! Well if this device is a router then something else is accessing the network beyond what I could see. I did a quick search on a MAC address search tool and found this result.


Apple? We don't have any Apple devices here at work. Or do we? What is made by Apple and often times accessing the network AND would be accessing the network BEFORE the user would login to his computer?


Yes, the iPhone. The user has an iPhone which was accessing the WiFi network. In order to access the WiFi network you need to login with your AD account and this is held on the iPhone. Here's is the login screen for the WiFi access.


Solution and closure

Once I removed this from the user's phone access from the WiFi, the lockouts stopped. When I tried to recreate this problem I couldn't seem to get my iPhone to lockout but the end user did say he had a sync software running on his iPhone with his desktop. I'm glad I found the problem and resolved it. This by far was the most difficult account lockout I have battled.

I hope this will be helpful for you!


MarcJ said...

Hi Rob,

This is a great article, and I'm sure must've already helped so many folks out there!

The Account Lockout Management tool from Microsoft certainly sounds quite helpful.

Speaking of which, do you happen to know of, or would you recommend any other free AD reporting tools that could help in finding locked accounts and doing similar useful things?

If so, I would be interested in learning more, as I run a blog on Free Active Directory Reporting Tools, and I would love to cover any such tools that you could suggest on it. (I will try to cover the Microsoft tool as well shortly.)

Thanks again, and nice work


- Marc

Rob said...

Hi Marc,

Thanks for the great comments. So far I have stuck with the Microsoft Lockout tools because they are simple to run, do not need any special installs or requirements, and they are (so far) all I've needed. In the case I wrote about it was a typical end user, who only logged into one workstation but if I was searching for an admin level account, who logged into many systems I might used another tool.

From the Microsoft Script Center, there are VBScripts, and Power Shell scripts that you can search a listing of workstations or servers and dump the service login information, but this is more of a script than actual tool.

I did make use of Sysinternals to find the process details, and NMap to scan ports but both of these are not AD tools. But each tool has it's purpose.

I'm going to follow your blog as it's very helpful for me and always looking for tools.

Please let me know if you have any other questions, will be glad to help out!


MarcJ said...

Hi Rob,

Thanks for following my blog. I appreciate it. I'm going to add your blog to my list of followed blogs as well.

If you have a Followed Blogs section on your blog as well, and would like to consider adding mine to yours, that would be mich appreciated.

Thanks, and ttyl.
- Marc

Rob said...

Will do!