Setting up managed code debugging (with SOS and SOSEX)
In this post I want to share some basics on how to set-up managed code debugging in windows debugger. I’ll consider that reader already know basics about how to setup windows debugging. I’ll mainly discuss details about what and how need to be loaded in order to be able to use specialized extensions for managed code debugging (mainly sos and sosex). I’ll not be discussing the debugging itself (e.g. commands from sos and sosex extensions).
Local live debugging
In the case of local live debugging we don’t have to much obstacles in our way (since we have access to the used .net framework binaries and data access layers – those I’ll discuss later). You should make sure that you are using same bitness of debugger and extension as is the bitness of the target application (.NET application are usually built architecture agnostic and then run as 32 bit processes on 32 bit OS and 64 bit processes on 64 bit OS).
If you would be about to use mismatching bitness of debugger to attach/start the managed process you might see the following error:
--------------------------- WinDbg:6.13.0014.1618 X86 --------------------------- Could not create process '\\Foo\Bar\Baz.exe', Win32 error 0n50 The request is not supported. --------------------------- OK ---------------------------
Once you are in the debugger, broken in into the target process, you will want to load sos extension (name is a acronym for ‘Son Of Strike’ as the extension was developed as the next version of the extension named ‘Strike’). This extension is shipped with .NET framework (and also indexed on Microsoft symbols server which will come in handy for other scenarios). An old version (1.1) of this extension also ships with the debugger package – and this can actually get into your way.
For .NET framework 1.1 the SOS extension has all the information about .NET internals ‘baked-in’ itself (some of those information you would normally expect to be in the symbol files) – so all you needed was loading the SOS. From the .NET version 2.0 the definition of data structures was separated into separate binary – mscordacwks (sometimes called data access layer). When loading the SOS extension it will need to also load correct data access layer (for correct architecture and version of .NET framework targeted by the debugged application/dump). Another important change for our purposes was introduced in .NET framework 4.0 – the mscorwks.dll module was replaced with module clr.dll. I’ll show how all those facts affects the managed code debugging.
Loading SOS extension during live local debugging
During the live local debugging you have the luxury of knowing that you computer has the same version and architecture of .NET framework as what is the debugged target using (because they are on the same machine :-)). Same approach also applies when you are debugging remotely or postmortem but know for sure that you have same architecture and version on your computer as what was on the target computer.
In this case you know that your installation of .NET framework will also have appropriate version SOS extension and mscordacwks.dll binary. They will actually in the same folder as where is the core dll for .NET – mscorwks.dll (for .NET up to 4.0) or clr.dll (for .NET 4.0 and higher) in that case you can simply use loadby command that will load an extension from a location of another binary that is already loaded in the debugger (so you need to make sure those modules are already loaded, or force load the with ‘.load /f <module>’):
.loadby sos.dll mscorwks .loadby sos.dll clr
Loading SOS extension during postmortem or remote debugging
If you would try previous commands on remote or postmortem debugging, you might happen to see the following error message:
The version of SOS does not match the version of CLR you are debugging. Please load the matching version of SOS for the version of CLR you are debugging. CLR Version: 4.0.30319.19010 SOS Version: 4.0.30319.269 Failed to load data access DLL, 0x80004005 Verify that 1) you have a recent build of the debugger (6.2.14 or newer) 2) the file mscordacwks.dll that matches your version of clr.dll is in the version directory 3) or, if you are debugging a dump file, verify that the file mscordacwks___.dll is on your symbol path. 4) you are debugging on the same architecture as the dump file. For example, an IA64 dump file must be debugged on an IA64 machine. You can also run the debugger command .cordll to control the debugger's load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload. If that succeeds, the SOS command should work on retry. If you are debugging a minidump, you need to make sure that your executable path is pointing to clr.dll as well.
The message itself already contains few helpful tips. However if you meet this situation you might be able to easily resolve majority of those cases by leveraging the fact that .NET Framework debugging related binaries are being indexed on Microsoft symbol server – internal and also external (for more info or symbol server, setting symbols path and execution image path you can refer to the setup windows debugging blog post). Basically for remote or postmortem debugging of managed application/dump in windows debugger you should follow those simple steps:
1) Set your symbol path to Microsoft symbol server (internal or external)
2) Set your execution image path to Microsoft symbol server
3) Make sure that the core CLR binary (mscorwks.dll or clr.dll) is loaded, reload it (with .reload /f) if it’s not
4) Reload CLR debugging related binaries by using ‘.cordll –ve –u –l’ command (-ve for verbose, –u for unload, –l for load)
5) Load the SOS module from the core CLR binary location (as you’d do that for live debugging)
Here is the example debugging session demonstrating those steps:
Microsoft (R) Windows Debugger Version 6.13.0014.1618 X86 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [\\XXXXXX\FooBar_121030_075015.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: ' *** Procdump -ma FooBar.exe *** Manual dump' Symbol search path is: YYYYYYYYY Executable search path is: Windows 7 Version 7601 (Service Pack 1) MP (32 procs) Free x86 compatible Product: Server, suite: TerminalServer DataCenter SingleUserTS Machine Name: Debug session time: Tue Oct 30 06:50:17.000 2012 (UTC - 8:00) System Uptime: 16 days 19:11:53.254 Process Uptime: 0 days 23:42:59.000 ................................................................ .................................................. Loading unloaded module list . Cannot Automatically load SOS eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=000001ec edi=00000000 eip=7784f8b1 esp=0044f164 ebp=0044f1d0 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 ntdll!ZwWaitForSingleObject+0x15: 7784f8b1 83c404 add esp,4 0:000> !eeversion No export eeversion found 0:000> .symfix+ 0:000> .sympath Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols Expanded Symbol search path is: srv*http://msdl.microsoft.com/download/symbols 0:000> .exepath SRV*http://msdl.microsoft.com/download/symbols Executable image search path is: SRV*http://msdl.microsoft.com/download/symbols Expanded Executable image search path is: srv*http://msdl.microsoft.com/download/symbols 0:000> lmvm mscorwks start end module name 709f0000 70f9b000 mscorwks (deferred) Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll Image name: mscorwks.dll Timestamp: Sat Dec 24 22:48:13 2011 (4EF6C72D) CheckSum: 005B474B ImageSize: 005AB000 File version: 2.0.50727.5456 Product version: 2.0.50727.5456 File flags: 0 (Mask 3F) File OS: 4 Unknown Win32 File type: 2.0 Dll File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® .NET Framework InternalName: mscorwks.dll OriginalFilename: mscorwks.dll ProductVersion: 2.0.50727.5456 FileVersion: 2.0.50727.5456 (Win7SP1GDR.050727-5400) FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation LegalCopyright: © Microsoft Corporation. All rights reserved. Comments: Flavor=Retail 0:000> .reload /f mscorwks.dll 0:000> lmvm mscorwks start end module name 709f0000 70f9b000 mscorwks (private pdb symbols) D:\Debuggers_32bit\sym\mscorwks.pdb\7139B75336C24F7CAA1DC4060608770D2\mscorwks.pdb Loaded symbol image file: mscorwks.dll Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll Image name: mscorwks.dll Timestamp: Sat Dec 24 22:48:13 2011 (4EF6C72D) CheckSum: 005B474B ImageSize: 005AB000 File version: 2.0.50727.5456 Product version: 2.0.50727.5456 File flags: 0 (Mask 3F) File OS: 4 Unknown Win32 File type: 2.0 Dll File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® .NET Framework InternalName: mscorwks.dll OriginalFilename: mscorwks.dll ProductVersion: 2.0.50727.5456 FileVersion: 2.0.50727.5456 (Win7SP1GDR.050727-5400) FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation LegalCopyright: © Microsoft Corporation. All rights reserved. Comments: Flavor=Retail 0:000> .cordll -ve -u -l CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.6387 f:0 doesn't match desired version 2.0.50727.5456 f:0 CLRDLL: Unable to find '' on the path Cannot Automatically load SOS CLRDLL: Loaded DLL D:\Debuggers_32bit\sym\mscordacwks_x86_x86_2.0.50727.5456.dll\4EF6C72D5ab000\mscordacwks_x86_x86_2.0.50727.5456.dll CLR DLL status: Loaded DLL D:\Debuggers_32bit\sym\mscordacwks_x86_x86_2.0.50727.5456.dll\4EF6C72D5ab000\mscordacwks_x86_x86_2.0.50727.5456.dll 0:000> .loadby sos mscorwks 0:000> !eeversion 2.0.50727.5456 retail Workstation mode SOS Version: 2.0.50727.6387 retail build
This should help you resolve vast majority of issues with loading modules for managed debugging, there still might be some special cases where you can see issues.
Issues with loading SOS extension and related binaries
One of those can happen when the mscordacwks is not properly indexed on the symols server. Then ouw would see similar output after running .cordll:
3: kd> .cordll -ve -u -l CLRDLL: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll:4.0.30319.269 f:8 doesn't match desired version 4.0.30319.19010 f:8 CLRDLL: Unable to find 'R?' on the path CLRDLL: Unable to find 'mscordacwks_AMD64_AMD64_4.0.30319.19010.dll' on the path CLRDLL: Unable to get version info for '\\ntdev.corp.microsoft.com\wincorerelease13\fbl_core1_kernel\9257.0.121021-1715\amd64fre\bin\indexes\..\netfx\core\win\microsoft.net\framework64\v4\mscordacwks_AMD64_AMD64_4.0.30319.19010.dll', Win32 error 0n87 CLRDLL: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_4.0.30319.19010.dll, Win32 error 0n87 CLR DLL status: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_4.0.30319.19010.dll, Win32 error 0n87
This can potentially happen to you inside Microsoft when working with specific build of .NET Framework that wasn’t properly indexed (or the index was already purged) – it should never happen outside of Microsoft. If this happens, contact CLR team, as they have their own share for the build binaries that might have longer data retention.
Other – most common – issue can happen when you accidentally load the improper version of sos (e.g. from the debugger package, by simply running ‘.load sos’ and picking some wrong version that is in the path). .chain command can help reveal those issues:
0:000> !eeversion Doesn't work with 2.x 0:000> .chain Extension DLL search Path: D:\Debuggers_32bit\WINXP;D:\Debuggers_32bit\winext;D:\Debuggers_32bit\winext\arcade;D:\Debuggers_32bit\pri;D:\Debuggers_32bit;D:\Debuggers_32bit\winext\arcade;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Odd\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;D:\Debuggers_64bit Extension DLL chain: D:\Debuggers_32bit\clr10\sos: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:41:23 2012 [path: D:\Debuggers_32bit\clr10\sos.dll] dbghelp: image 6.13.0014.1618, API 6.2.6, built Wed Oct 17 10:42:02 2012 [path: D:\Debuggers_32bit\dbghelp.dll] ext: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:41:49 2012 [path: D:\Debuggers_32bit\winext\ext.dll] exts: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:42:19 2012 [path: D:\Debuggers_32bit\WINXP\exts.dll] uext: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:42:18 2012 [path: D:\Debuggers_32bit\winext\uext.dll] ntsdexts: image 6.2.9255.0, API 1.0.0, built Wed Oct 17 10:42:28 2012 [path: D:\Debuggers_32bit\WINXP\ntsdexts.dll]
Loading SOSEX extension
SOSEX is helpful extension that extends possibilities of SOS. You first need to have SOS loaded – as described above. Then you can simply download latest version of SOSEX from http://www.stevestechspot.com/ and load it into the debugger and use it:
0:000> .load D:\Tools\sosex\32bit\sosex This dump has no SOSEX heap index. The heap index makes searching for references and roots much faster. To create a heap index, run !bhi 0:000> !sosex.help SOSEX - Copyright 2007-2012 by Steve Johnson - http://www.stevestechspot.com/ To report bugs or offer feedback about SOSEX, please email firstname.lastname@example.org Quick Ref: -------------------------------------------------- bhi [filename] BuildHeapIndex - Builds an index file for heap objects. bpsc (Deprecated. Use !mbp instead) chi ClearHeapIndex - Frees all resources used by the heap index and removes it from memory. dlk [-d] Displays deadlocks between SyncBlocks and/or ReaderWriterLocks dumpgen <GenNum> [-free] [-stat] [-type <TYPE_NAME>] Dumps the contents of the specified generation [-nostrings] finq [GenNum] [-stat] Displays objects in the finalization queue frq [-stat] Displays objects in the Freachable queue gcgen <ObjectAddr> Displays the GC generation of the specified object gch [HandleType]... Lists all GCHandles, optionally filtered by specified handle types help [CommandName] Display this screen or details about the specified command lhi [filename] LoadHeapIndex - load the heap index into memory. mbc <SOSEX breakpoint ID | *> Clears the specified or all managed breakpoints mbd <SOSEX breakpoint ID | *> Disables the specified or all managed breakpoints mbe <SOSEX breakpoint ID | *> Enables the specified or all managed breakpoints mbl [SOSEX breakpoint ID] Prints the specified or all managed breakpoints mbm <Type/MethodFilter> [ILOffset] [Options] Sets a managed breakpoint on methods matching the specified filter mbp <SourceFile> <nLineNum> [ColNum] [Options] Sets a managed breakpoint at the specified source code location mdso [Options] Dumps object references on the stack and in CPU registers in the current context mdt [TypeName | VarName | MT] [ADDR] [Options] Displays the fields of an object or type, optionally recursively mdv [nFrameNum] Displays arguments and locals for a managed frame mfrag [-stat] [-mt:<MT>] Reports free blocks, the type of object following the free block, and fragmentation statistics mframe [nFrameNum] Displays or sets the current managed frame for the !mdt and !mdv commands mgu // TODO: Document mk [FrameCount] [-l] [-p] [-a] Prints a stack trace of managed and unmanaged frames mln [expression] Displays the type of managed data located at the specified address or the current instruction pointer mlocks [-d] Lists all managed lock objects and CriticalSections and their owning threads mroot <ObjectAddr> [-all] Displays GC roots for the specified object mt (no parameters) Steps into the managed method at the current position mu [address] [-s] [-il] [-n] Displays a disassembly around the current instruction with interleaved source, IL and asm code muf [MD Address | Code Address] [-s] [-il] [-n] Displays a disassembly with interleaved source, IL and asm code mwaits [-d] Lists all waiting threads and, if known, the locks they are waiting on mx <Filter String> Displays managed type/field/method names matching the specified filter string refs <ObjectAddr> [-target|-source] Displays all references from and to the specified object rwlock [ObjectAddr | -d] Displays all RWLocks or, if provided a RWLock address, details of the specified lock sosexhelp [CommandName] Display this screen or details about the specified command strings [ModuleAddress] [Options] Search the managed heap or a module for strings matching the specified criteria ListGcHandles - See gch Use !help <command> or !sosexhelp <command> for more details about each command.
That’s it today. Now you are introduced to setting up managed code debugging and next time I’ll try to post some real live interesting case scenario.