HTTP.SYS / Cryptographic Services / LSASS.EXE deadlock

NOTE: This issue has been subsequently fixed in KB 2379016 A computer that is running Windows Vista or Windows Server 2008 stops responding at the "Applying User Settings" stage of the logon process.
(This blog entry is left for reference.)

A recent case I had brought this issue to my attention, so I thought it useful to share the knowledge…

The problem encountered was a Windows Server 2008 x64 SP2 server running several websites was failing to start several services during startup, and attempts to logon stuck at “Applying User Settings… ” indefinitely.
Starting in Safe Mode (or Safe Mode with Networking) worked fine.

By the time the case had been opened the symptom had already been removed by taking away some certificates – the server then started correctly when booted normally (adding re-adding the certificates did not reintroduce the problem).

 

From a VM image of the server in the problem state I made a complete memory dump when the server had been stuck ~10 minutes during user logon.

From the hang dump I could see the logon was stalled because one of the threads in LSASS.EXE was waiting for the Cryptographic Services service to start – an ALPC was sent to to the Service Control Manager (SCM) to poke the thread when the service was up.

SCM was in the process of starting HTTP.SYS, which held the lock for service startup (preventing the Cryptographic Services service from starting) and HTTP.SYS’s thread requires the services of LSASS.EXE…deadlock.

http-lsass-cryptsvc_deadlock

This is a classic timing issue – HTTP.SYS is slow to load (for some reason relating to the SSL binding information) and is holding the SCM lock long enough to stumble into the “Mexican stand-off”.

We can iron out the kink by putting a dependency for HTTP.SYS on Cryptographic Services (CryptSvc), so it will always wait, so when it makes a call into LSASS.EXE it is not holding the lock preventing LSASS.EXE’s dependency from starting.

This workaround is described here: Computer hangs at Applying Computer Settings or All Automatic Services Will Not Start After Reboot on Windows Server 2008

 

Does this mean that the symptom of “Automatic services failing to start” are caused by this one issue?
NO

The symptom is unfortunately very generic – there is so much going on with a system startup that there are many ways to encounter a “hang” with totally different root causes.

It is very easy to say “I am experiencing the symptoms described in KB article XXX, but the solution/workaround did not work” – this invariably means that you did not have that problem described by the article.

But if you have a server with certificates installed and SSL is configured (most probably for use by IIS), and you’re running Windows Server 2008 then it’s worth knowing about this issue and trivial workaround.

(Windows Server 2008 R2 does not have the problem, by the way.)