Kernel Debugging Day 1 - Interesting snippets of information for debugging in the trenches

Blue Male Doctor In A Lab Coat, Sitting On A Stool And Bandaging A Blue Person That Has Been Hurt On The Head, Arm And Ankle Clipart GraphicContext

Two weeks ago an invite went out for a “Kernel Mode Debugging” course and as I am always interested in debugging and a challenge, I signed up.

That’s where I was today and for the next two days. I have not done any kernel mode development for some time and am not planning to for the foreseeable future … so why am I here? Well, as mentioned, I love the challenge of exploring new things, but more importantly I have always been interested in being able to ponder over a “blue screen” and then be able to analyse and “kind-of-determine” what “could” have happened. 

By the way, if you are interested in debugging you must subscribe to this blog: If broken it is, fix it you should!

Some random, but interesting information I made a note of today
Blue Man Holding a Pencil and Drawing a Circle on a Blueprint Clipart Illustration

Did you know … that:

  1. User mode stack is 1MB by default and kernel mode stack is limited to 12k?
  2. User mode knows a heap and kernel mode a pool?
  3. User mode has the AppVerifier and kernel mode the DriverVerifier?
  4. user mode is at home between addresses 00000000h and 7FFFFFFFh, while kernel mode lives and breathes between 800000000h and FFFFFFFFh?

How are crash dumps created?
Blue Man Leaning Against a Stack of Papers Clipart Illustration

My focus, as mentioned, is not on driver development and debugging, but trying to understand kernel mode crash dumps.

What types of crash dumps do we have?image

  • Small Dump … limited crash dump containing context and stack information for the faulting thread, current tread and process structures, kernel mode list, memory management data structures and pseudo-random pages related to the crash.
  • Kernel Dump … contains resident kernel pages only and NO user mode stack or data. This type is good for blue screen analysis as most blue screens are hardware, driver and kernel mode related.
  • Full Dump … contains the entire physical memory, and both kernel and user mode resident pages. Caution when running on 64-bit systems with lots and lots of RAM … the crash dump file will be huge!

How and when are they created?

  • First we have to configure the system to write a crash dump … see Start-up and Recovery dialog on the right.
  • Important Notes:
    • Crash dumps created soon after system boot can be compressed much better, as the memory space and therefore the crash dump are primarily zeroes. Over time, memory is filled with data and becomes less compressible.
    • The page file must be large enough to accommodate the crash dump, else no crash dump will be created.
    • If you need a tool to force a crash dump, have a look at Bang on Bing.
  • When the system crashes, the physical memory is written to the page file … based on the type of crash dump selected this can range from a few kilobytes to Terabytes on a 64-bit machine.
  • When the system reboots, a special program (savedump.exe) copies the page file contents to the crash dump file.

… so now that we have the basics of crash dumps, what is possible when we are staring at a crash dump file. Today we covered the following scenarios, each of which I will cover briefly. A more detailed walkthrough of these scenarios are beyond the scope of this blog or my understanding … if needed, you best refer to books such as “Advanced Windows Debugging ISBN:0321374460” or “Memory Dump Analysis Anthology”.

Some of the Important Commands
Clipart Illustration of a White Character Taking Notes Down On A Clipboard, A Supervisor, Manager, Or Someone Taking Inventory 

Command with an exclamation mark (!) in front are extensions to WinDBG. To display the extension DLLs, run the command:

kd> .chain
Extension DLL search Path:
c:\windbg\WINXP;c:\windbg\winext;c:\windbg\winext\arcade;c:\windbg\pri;c:\windbg;c:\windbg\winext\arcade;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\Windows Imaging\;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Microsoft Magellan Code Coverage Toolset\;c:\Program Files\System Center Operations Manager 2007\;C:\Program Files\Gumshoe\
Extension DLL chain:
    dbghelp: image 6.11.0001.404, API 6.1.6, built Wed Feb 25 17:55:30 2009
        [path: c:\windbg\dbghelp.dll]
    ext: image 6.11.0001.404, API 1.0.0, built Wed Feb 25 17:55:30 2009
        [path: c:\windbg\winext\ext.dll]
    exts: image 6.11.0001.404, API 1.0.0, built Wed Feb 25 17:55:24 2009
        [path: c:\windbg\WINXP\exts.dll]
    kext: image 6.11.0001.404, API 1.0.0, built Wed Feb 25 17:55:24 2009
        [path: c:\windbg\winext\kext.dll]
    kdexts: image 6.1.7015.0, API 1.0.0, built Wed Feb 25 17:54:56 2009
        [path: c:\windbg\WINXP\kdexts.dll]

To get help for the extensions, run the following command:

kd> !
acpicache [flags]      - Displays cached ACPI tables
acpiinf                      - Displays ACPI Information structure

… outout clipped …


description from WinDBG Help (go there for detailed help!)

!analyze -v displays information about the current exception or bug check.
!devstack displays a formatted view of the device stack associated with a device object.
!drvobj displays detailed information about a DRIVER_OBJECT.
!handle displays information about a handle or handles that one or all processes in the target system own.
!irql displays the interrupt request level (IRQL) of a processor on the target computer before the debugger break.
!memusage displays summary statistics about physical memory use.
!poolused displays memory use summaries, based on the tag used for each pool allocation.
!podev displays the power capabilities of the target computer.
!process displays information about the specified process, or about all processes, including the EPROCESS block.
!pte displays the page table entry (PTE) and page directory entry (PDE) for the specified address.
!ready displays summary information about each thread in the system in a READY state.
!session displays one or more user sessions, or displays a specified process running in multiple user sessions.
!stacks displays information about the kernel stacks.
!teb displays a formatted view of the information in the thread environment block (TEB).
!thread displays summary information about a thread on the target system, including the ETHREAD block.
!vm displays summary information about virtual memory use statistics on the target system.
.reload deletes all symbol information for the specified module and reloads these symbols as needed
.sympath changes the default path of the host debugger for symbol search.
.trap displays the trap frame register state and also sets the register context.
dps The dds (double word), dps (pointer-sized), and dqs (quad-word) commands display the contents of memory in the given range.
kvf, kvn display the stack frame of the given thread, together with related information
lm displays the specified loaded modules.
r displays or modifies registers, floating-point registers, flags, pseudo-registers, and fixed-name aliases.
ub displays an assembly translation of the specified program code in memory.
uf displays an assembly translation of the specified function in memory.
up displays an assembly translation of the specified program code in physical memory.
vertarget displays the current version of the Microsoft Windows operating system of the target computer.

How do we analyse these crash dumps?
Pissed Blue Person Breaking A Flat Screen Computer Monitor With A Hammer Clipart Illustration Graphic

  • ---------------------------------------------------
    Scenario 1 – Light up the driver
    1. Main commands used:
      • !analyze –v
      • dps
      • lmvm
      • .sympath
    2. Extract of crash dump analysis:
    3. Steps to crash dump analysis:
      • {line:35} We run dps on the work item address as hinted in {line:9,10}. We notice a third-party driver {line40} … the less than friendly driver+address format gives it away.
      • {line:68} lmvm gives us more detail for the cdp driver.
      • {line:78} We load the symbols, by running .sympath
      • {line:83} Running dps again, shows us that the driver has lit up and we even see the offending method.
  • ---------------------------------------------------
    Scenario 2 – Find the module responsible for a crashClipart Illustration of a White Character Inspecting A Laptop Computer With A Magnifying Glass, Over White
    • Main commands used:
      • !analyze –v
      • !devstack
      • !podev
    • Extract of crash dump analysis:
    • Steps to crash dump analysis:
      • {line:24} !analyze –v creates the analysis for easier reading
      • {lines:31-34} The analysis pretty much tells us what the problem is, how to fix it in the driver and who the offending driver is … but for the latter, let’s do a few more steps.
      • {line:92"} !devstack 814a32d0 gives us the device stack and we can see that NTPNP_PCI0005 is the bus driver (bottom) and the others drivers. By the way, we got the address from {line:34}.
      • {lines: 101,109,117 and 125} !podev shows us the power abilities for each driver.
      • {line:107} We notice the invalid power flags, highlighted in yellow and assume that the xyzHostLegacyVideo driver is the area of concern.
      • {line:33} Points us to the same driver … voila, we know who to call.
  • ---------------------------------------------------
    Scenario 3 – Identify the cause of a driver crash
    • Main commands used:
      • !analyze –v
      • !drvobj
      • !irp
    • Extract of crash dump analysis:
    • Steps to crash dump analysis:
      • {line:17} analyze –v creates the analysis for easier reading
      • {lines 50-60} quite clearly point us towards the dummydma driver as the problem child
      • {line:62"} !drvobj abcdma 2 gives us the dispatch routines for the driver.
      • Option A
        • {line:34} Defines the return address as f90a8799.
        • {lines:94-99} We calculate the address as 799. Looking at the dispatch table we note that IRP_MJ_CLEANUP is closest to this address and hence the likely candidate.
      • Option B – Oh, Oh
        • {line:100} We assume that the second parameter is an IRP (I/O request packet) after reviewing the IoCallDriver method. The !irp 80068f7c command fails?!?
        • {line:37} Note the f, which indicates fast calling method. This means that the IRP is passed in registers, not the call stack.
      • Option C
        • {line:103} We assume the second parameter to the dummydma call is the IRP and call !irp ffb31d48
        • {line:109} The secret is out, the IRP specifies the 12th dispatch routine, which is indeed IRP_MJ_CLEANUP.
      • Option D
        • {line:114} !thread gives us the running thread and on {line:117} the same IRP.

Last but not least …

Just when I felt like one after a head-spinning day, I found this command:

  • !zombies This extension displays all dead ("zombie") processes or threads.