Dump Analysis in a Disconnected Environment

  • Article

Over the years I have come across the following restrictions in a variety of environments:

  1. The machines that we could do dump analysis did not have access to the internet.
  2. The dump could not leave those machines.

These types of restrictions is a common reason field team members are brought onsite since they cannot ship data offsite.  The lack of internet access though raises the question - “How do I get the symbols I need?”

The first option is to use the Symbol Packages that Microsoft provides here - https://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx

Now these symbol packages are a great starting point but they have a few key limitations:

  1. They are only for Windows.  So if I need .NET symbols and such I will be out of luck.
  2. They are only for major release milestones.  This means that if you have system that is patched you will find at least some symbols do not match.  While this will not be a problem in all debug scenarios it will be a problem in certain cases.

For both of these situations the Microsoft Public Symbol Server (https://msdl.microsoft.com/download/symbols) addresses this because the symbols are more frequently updated and also allows you to get symbols for more Microsoft products than just the OS.  The rub is - “How do I know what symbols to pull down if I cannot get the dump to the internet?”

Enter SYMCHK - https://msdn.microsoft.com/en-us/library/cc267474.aspx

This is a utility to validate symbols.  It has a “Manifest” feature that we can utilize here.  The idea is that create a series of steps like this:

  1. Generate a Manifest.
  2. Move that manifest to another machine.
  3. Create a Symbol Cache Using the Manifest.
  4. Move the Symbols to the Closed Machine.

Let’s look at each of these steps in a bit more detail.

Generate a Manifest

To do this you can just run the following command:

D:\debuggersx86>symchk /id D:\DumpFiles\Dump1.dmp /om symbol.txt

SYMCHK exists in your the directory where you installed the Debugging Tools for Windows.  You can replace the DMP file above with the location of the dump file that you wish to debug.  The “symbol.txt” file is the manifest that we are generating.

The resultant manifest will contain a line for each module and look like this:

kernel32.dll,46239bd5f5000,2
ntdll.dll,411096b4b0000,2

Moving the Manifest

This is probably the trickiest part depending on the place where you work.  The idea here is that the file you have to move is simply a text file of file names.  You can do several things:

  1. Scrub out any custom modules so those do not leave.
  2. Write down the contents of the file if need be.

The net of this is that you have much more limited set of information to move off the system as opposed to moving a dump or even minidump which might be harder in various areas.

Creating a Symbol Cache Using the Manifest

After you have the manifest on a machine with internet access you can then sync down the symbols using a command like this:

symchk /im symbols.txt /s srv*d:\tempSym*https://msdl.microsoft.com/download/symbols

You can use any directory in place “d:\tempSym”. 

Move the Symbols to the Closed Machine

Once the symbols have come down just copy the cache directory that you used in the previous step (d:\tempSym in the example) and move that over to the closed machine.  Then in the debugger set your symbol path to include that folder. For instance if you copied the folder to d:\tempSym on the close machine you would add “srv*d:\tempSym” to your symbol path to start loading symbols from the folder.