Capturing memory dumps for 32-bit processes on an x64 machine
This is an issue that I often get questions around and we often have cases where we have to re-capture memory dumps because the memory dumps were captured the “wrong” way.
The short story is: If you are executing a 32-bit process on a 64-bit machine, (which is the default case for IIS on x64 machines for example) you need to capture the dump with a tool that allows you to create 32-bit dumps.
How do you know if your process is 32-bit?
If you are on a 64-bit machine, you can check task manager to see what architecture your process is using.
Processes with *32 are 32-bit and the rest are 64-bit so in the example above we can see that for example w3wp.exe is executing 32-bit code.
Why is it important to capture them with the right tools?
If you capture a dump with a tool that captures 64-bit dumps you will still get a memory dump, but you will get a memory dump of the syswow64 which means that a lot of extensions like sos and psscor2 won’t be able to read the data. Some things may still work but it is very limited and you might get errors or erroneous call stacks etc.
Typical stuff that happens when you try to read a 64-bit memory dump of a 32-bit process is:
1. You may see warnings like
WARNING: wkscli overlaps srvcli ..............WARNING: wship6 overlaps dnsapi .WARNING: IPHLPAPI overlaps dsrole ...WARNING: FWPUCLNT overlaps rasadhlp WARNING: FWPUCLNT overlaps dnsapi .....WARNING: compstat overlaps iisres
2. The stacks show wow64cpu methods
0:000> kp Child-SP RetAddr Call Site 00000000`000ce728 00000000`73a22bcd wow64cpu!CpupSyscallStub+0x9 00000000`000ce730 00000000`73a9d07e wow64cpu!Thunk0ArgReloadState+0x1a 00000000`000ce7f0 00000000`73a9c549 wow64!RunCpuSimulation+0xa 00000000`000ce840 00000000`76d684c8 wow64!Wow64LdrpInitialize+0x429 00000000`000ced90 00000000`76d67623 ntdll!LdrpInitializeProcess+0x17e2 00000000`000cf290 00000000`76d5308e ntdll! ?? ::FNODOBFM::`string'+0x2bea0 00000000`000cf300 00000000`00000000 ntdll!LdrInitializeThunk+0xe
3. You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process.
4. You get errors like the below when trying to run sos commands.
(Note: You can get these errors if you don’t have symbols properly set up as well, so this is not the only reason for getting these errors)
0:000> !eeheap -gc Failed to load data access DLL, 0x80004005 Verify that 1) you have a recent build of the debugger (6.2.14 or newer) 2) the file mscordacwks.dll that matches your version of clr.dll is in the version directory 3) or, if you are debugging a dump file, verify that the file mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path. 4) you are debugging on the same architecture as the dump file. For example, an IA64 dump file must be debugged on an IA64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
What tools should you use?
The best way to capture a process dump in this scenario is to use 32-bit versions of tools like Debug Diag, or adplus+cdb (32 bit debugging tools for windows). You can install the 32-bit versions of these tools on a 64-bit system. In the next version of Debug Diag you will be able to capture 32-bit dumps with the 64-bit version of Debug Diag.
I have mentioned before that you can capture memory dumps with task manager on Vista+, but if you do this on a 64-bit machine you will get a 64-bit dump. You can however use the 32-bit task manager, located in C:\Windows\SysWOW64\taskmgr.exe to get 32-bit dumps. If you want to verify that you are running the 32-bit version of task manager, you can check that taskmgr.exe is listed with *32 in task manager itself.
Edit: Sorry Mark and David, I of course didn't mean to leave out the nice and very lightweight procdump tool from SysInternals which will capture 32-bit dumps of 32-bit processes even on 64-bit Windows. Thanks for reminding me:)
For some reason I thought that I had written about this tool in an earlier post but a quick search reminded me that this is still in my blog-todo bin.
For all other dump capturing tools, I would suggest that you read the documentation to see if there is a 32-bit version (like for procmon where you can pass in the /run32 parameter) or if they will capture 32-bit dumps of 32-bit processes.
Is it important to read the dump with the right “bit” debugger?
Yes, to read 32-bit dumps you should use 32-bit windbg (either on a 32-bit or 64-bit OS), and for 64-bit dumps, the machine that you debug the dumps on, needs to be a 64-bit machine, running 64-bit windbg.
Remember that you should also use the 64-bit versions of sos, psscor2, sosex and any other extensions that you might use, if you are debugging a 64-bit dump.
Have a good one,