Data Collection for Deadlock Detected issue of a IIS based web application
Defining a w3wp deadlock detected issue:
Sometimes we might see that the website hosted on IIS became unresponsive for some time and we see any of the below event log entry
1) ISAPI 'C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.
2) ISAPI’C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_isapi.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected’”
3) Description: ISAPI 'C:\Windows\system32\inetsrv\asp.dll' reported itself as unhealthy for the following reason: 'ASP unhealthy because 100% of executing requests are hung and 0% of the request queue is full.'
When a Web application is busy, there may be instances when the maximum number of threads has been spawned and some of the threads are unresponsive, resulting in degraded performance. IIS has the ability to solve the problem of unresponsive threads by recycling the worker process that hosts that particular instance of the ISAPI. When threads are unresponsive the appropriate ISAPI i.e ASP.dll or aspnet_isapi.dll calls the ISAPI server support function HSE_REQ_REPORT_UNHEALTHY, and the WAS (in IIS 7) or WWW service(in IIS 6) will recycle the worker process and make an entry in the event log.
So when we see a deadlock there might not be an actual deadlock for any resource and it might just be that all the threads are busy doing some task.
Steps to Create Debug Diag Rule:
1. We will use Debug Diagnostic 2.0 update 2 version to troubleshoot this issue further. Please install the Debug Diagnostic 2.0 update 2 tool from https://www.microsoft.com/en-us/download/details.aspx?id=49924 (Please un-install all other versions of Debug Diagnostic tool before installing the latest version)
2. Before proceeding with the rule creation we need to download the symbols needed for the automated rule creation. Take the below steps to download the symbols for your w3wp process.
|a. Download PDBDownloader.exe from https://github.com/rajkumar-rangaraj/PDB-Downloader/releasesb. Run the PDBDownloader.exe and click on open files
c. If it is a IIS 6 machine, Navigate to the path C:\Windows\System32\inetsrv\ and select w3isapi.dll. Click on Start
d. If it is a IIS 7 or higher machine, Navigate to the path C:\Windows\System32\inetsrv\ and select isapi.dll. Click on Start
e. If it is a 32 bit application on a 64 bit IIS 6 machine, Navigate to the path C:\Windows\SysWow64\inetsrv\ and select w3isapi.dll. Click on Start
f. If it is a 32 bit application on a 64 bit IIS 7 or higher machine, Navigate to the path C:\Windows\System32\inetsrv\ and select isapi.dll. Click on Start
g. The tool will download the public symbols of w3isapi.dll or isapi.dll to C:\Symbols folder and we are good to proceed with the rule creation
|Debug Diag 2.0 Update 2 uses the symbol folder as C:\Symbols. Debug Diag 1.2 or Debug Diag 2.0 Update 1 uses the symbol folder as C:\symcache. If you have Debug Diag 1.2 or Debug Diag 2.0 Update 1 then change the file location in PDBDownloader.exe to C:\symcache. If folder does not exist please create it.|
|Note: The executable requires internet connection to download the symbols. If you don’t have internet connection on the server, then copy the C:\Windows\System32\inetsrv\w3isapi.dll or C:\Windows\System32\inetsrv\isapi.dll to the machine which has internet connection and run the tool over there. Once you have the symbols downloaded, copy PDB file to the server’s symbol folder. In case of Debug Diag 1.2 or Debug Diag 2.0 Update 1copy it to C:\Symcache, else use C:\Symbols|
3. Create a Crash Rule in Debug Diagnostic 2.0 with the following steps:
a. Open Debug Diag (Start -> Programs -> Debug Diagnostic Tool 2.0 Collection)
b. Select "Crash" and click Next
c. Choose the option “A specific IIS application pool” and click Next and choose the application pool that is crashing from the list and click Next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)
d. Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint
e. If it is a IIS 7 or higher machine then addisapi!SSFReportUnhealthy and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok.
If it is a IIS 6 machine then Add w3isapi!SSFReportUnhealthy and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok.
f. Click on Save and Close
g. Click Next for "Rule Name". The "User dump Location" can be changed here.
h. Select "Activate the rule now" and click Finish
4. Notice the Status is Active. The User dump Count will increase each time a dump file is created.
5. Wait for the issue to happen.
Once the issue occurs
1. The user dump count field in the Crash Rule will increment whenever the tool generates a dump. The dump will be generated in the User Dump Path
2. You can also analyze the Dump by running Debug Diag Analysis Tool and clicking Add Data Files. When the .dmp is added, select the Crash/Hang Analyzers script and click Start Analysis. When finished, a report (.mht) will be created in C:\Program Files\DebugDiag\Reports and displayed in Internet Explorer with the results and recommendations. If using custom DLL’s, the Symbol path (Tools menu -> Options and Settings -> Symbol Search Path) to the custom PDB files can be added.
Hope this helps