Finding locked out users in AD and their source

Few days ago we received a complaint from a user that his account is constantly getting locked. As many of you know AD can be configured to lock user accounts if users enter their password wrong too many times in specified time. This prevents brute force attacks but can expose your environment to denial of service attacks. Although the most common case for this to occur is that the user change their password and forget to change it to mobile phone which is used to check emails or other devices storing the username and password for authentication.

Usually security log on domain controllers contain the logon workstation name for those failed authentication attempts to help identify the source but sometimes that is not enough. Like in our case the name of the logon workstation in the log entry was \\workstation. That didn't help us much since we do not have any workstations with that name in our domain and our network discovery tool didn't show anything with that name either. To find out the source we had to combine few things which I explain in this blog post.

1. Find out currently logged users in domain for example using powershell:

Import-Module ActiveDirectory
Search-ADAccount -LockedOut | Format-Table -Property samaccountname -AutoSize -HideTableHeaders

2. Use these samaccountnames to find out the domain controller which is receiving these authentication requests. This can be achieved by using powershell to query all domain controllers for LastBadPasswordAttempt:

Get-ADDomainController -Filter { domain -eq $yourdomainname } | foreach { write-output "$($_.hostName) says last bad password attempt for $($samaccountname) was: $((Get-ADUser $samaccountname -Server $_.name -Properties LastBadPasswordAttempt).LastBadPasswordAttempt)" }

where $yourdomainname is the canonical name of your domain, for example corp.example.com and $samaccountname is the account that is locked out.

When the DC which is receiving those incorrect credentials is found you can enable netlogon debug logging by typing:

nltest /dbflag:0x2080ffff

to an administrative command prompt on that DC.

After running that command you will have to restart netlogon service for the change to take effect. The easiest way to do this is simply typing:

net stop netlogon && net start netlogon

to a command prompt. After you have restarted netlogon service you can check the %Systemroot%\Debug\Netlogon.log file to trace the account lockout to the source where incorrect credentials were used. Help for analyzing the netlogon debug logs and error codes can be found at http://technet.microsoft.com/en-us/library/cc776964%28WS.10%29.aspx

After you have found the source for locked user account the netlogon debug flag should be disabled using command:

nltest /dbflag:0x0

And restarting netlogon service again. For curious people out there, those nltest commands modify HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\DBFlag registry entry to specified value. More details about enabling netlogon debug can be found at http://support.microsoft.com/kb/109626

That's all this time. Next time it is much easier to find out where the bad password attempts are originating and resolve the issue.

I'm back.. :D to write about Event ID 2213 in DFS Replication log on Windows Server 2012 Domain Controller

Wow, it has been a long time since last post. Almost two years. Been too "busy" to write anything. About the time to correct that.

Today I came across with DC restarting itself unexpectedly. Still wondering why but that is another story. That unexpected restart caused dirty shutdown of DFSR JET Database. Default behavior of Windows Server 2012 changed from 2008 automatic recovery to manual recovery.

Okay, fine. Now we have chance to take backup of our existing replicating folders before the autorecovery might merge them so that the winning files are not those we want. Sounds great. On the other hand replication is stopped on that volume until someone manual tells it to resume. Not so great if that volume happens to be the volume hosting SYSVOL.

Well, now you think that why should we care, we have Operations Manager monitoring our AD replication and we notice immediately that replication is not working and can manually resume it. One thing that nobody tell you is that DFS Replication monitoring is watching for event id 2212 that states:

"The DFS Replication service has detected an unexpected shutdown on volume %2. This can occur if the service terminated abnormally (due to a power loss,for example) or an error occurred on the volume. The service has automatically initiated a recovery process. The service will rebuild the database if it determines it cannot reliably recover. No user action is required."

The catch is that on Windows Server 2012 DCs what you get is event id 2213 when autorecovery is not on. That event description states that:

"The DFS Replication service stopped replication on volume %2. This occurs when a DFSR JET database is not shut down cleanly and Auto Recovery is disabled. To resolve this issue, back up the files in the affected replicated folders, and then use the ResumeReplication WMI method to resume replication."

That event id 2213 in DFS Replication log from DFSR source is NOT monitored by default on SCOM 2012 AD management pack. Windows Server 2012 is by the way categorized still as 2008.

Luckily it is easy to implement your own monitor to trigger alert when event id 2213 is seen and automatically close the alert when event id 2214 is recorded.

What makes this even more interesting is that according to http://support.microsoft.com/kb/2846759 recommended best practice for Windows Server 2012 is that this autorecovery should be turned on. Maybe someone responsible for AD management pack didn't get that memo about that original design change or forgot to include event id 2213 as a trigger to raise an alert about DFS replication issues. Hopefully this gets fix in the future management packs.

Exchange 2010 SP2 makes hybrid deployment with Office 365 easier

Exchange 2010 SP2 was released earlier this week and one of the new features seems very interesting for those who are planning hybrid deployment of Exchange 2010 with Office 365.

Exchange 2010 SP2 introduces the hybrid configuration wizard which replaces approximately 50 of the manual steps to configure hybrid deployment with a few inputs and just six clicks of your mouse.

More information about the new wizard can be found from Technet.

I have to definitely try it myself when I get some spare time to play around with SP2 and Office 365 again.

Technical documentation for Windows Server Protocols and authentication 101

Yesterday I rediscovered a great resource about Windows Server protocols when I was debugging SMB2 and Kerberos V5 traffic between Windows 7 client, Netapp filer and DC on Windows Server 2008 R2.

I found out a page on MSDN that contains the technical documents for the Windows Server Protocols (WSPP) documentation set. Especially [MS-SPNG]: Simple and Protected GSS-API Negotiation Mechanism (SPNEGO) Extension, [MS-KILE]: Kerberos Protocol Extensions and [MS-SMB2]: Server Message Block (SMB) Version 2 Protocol Specification

I also found helpful blog post on Microsoft Open Specifications Support Team Blog explaining authentication in Windows.

FIM 2010 R2 Release Candidate available

Forefront Identity Manager 2010 R2 Release Candidate is announced. FIM 2010 R2 RC can be downloaded from Connect web site. Technical information and release notes are available in TechNet Library.

64-bit DirSync now available from Office 365 portal

64-bit version of Directory Synchronization tool is released. 64-bit DirSync is based on FIM 2010 when the old 32-bit version was based on ILM. You need to have 64-bit standard or enterprise edition of Windows Server 2008 or Windows Server 2008 R2. More information can be found from Office 365 blog.

Security Compliance Manager version 2.0.20.0 released

Long time no post... No explanations this time. ;)

Microsoft Security Compliance Manager version 2 has been released few weeks ago. Take a look at it if you want to access and automate all of your organization’s security baselines in one centralized location.

The tool allows you to implement security baselines for Windows Server 2008 R2 SP1, Windows Server 2008 SP2, Windows Server 2003 SP2, Hyper-V, Windows 7, Windows Vista SP2, Windows XP SP3, BitLocker Drive Encryption, Windows Internet Explorer 9, Windows Internet Explorer 8, Microsoft Office 2010, and Microsoft Office 2007 SP2. Tool allows you to compare your own production GPOs against Microsoft's guidance. You can export either tested Microsoft's baselines or your own customized baselines directly to group policies which you can import to your AD.