Tools of the Trade: Part IV – WinDbg & AdPlus (I)
In this series of posts, we’re going to take a look at some of the tools we at Microsoft Developer Support -- Database commonly use, how to use them, and the types of issues they can be used to resolve.
In this article, Enamul Khaleque starts down the road of using WinDbg and AdPlus. Future articles will move beyond identification, and discuss step-by-step approach of using WinDbg to solve wide array of issue including but not limited to: memory leaks, high & low CPU hangs, heap corruption, race conditions and crashes/exceptions in both managed and unmanaged environments.
You may be wondering why I am talking about WinDbg and AdPlus while you already have Visual Studio debugger. If you ask your network/system admin to install Visual Studio in a production environment, the odds of hearing a big NO is 99.99% if not 100%. That’s because Visual Studio is an extensive development environment and not suited for a production environment. Fortunately, we can capture runtime state in a file (called a memory dump or simply dump) using light-weight tools such as AdPlus and later analyze them using WinDbg. For this article, I will assume you have no prior experience in WinDbg or AdPlus.
WinDbg and AdPlus are part of Debugging Tools for Windows. Both 32-bit and 64-bit versions are downloadable from following URL:
Imagine that after upgrading your application in production, you are now seeing significant response delay. You are not able to reproduce this behavior in a Dev, Test or QA environment, so your only option is to capture the application state while the latency is occurring at runtime. With AdPlus, you can capture a “hang mode” memory dump to determine what the process is waiting on.
Instead of seeing delays, let’s say this application is crashing in the production and you can’t determine why. With AdPlus, you can also capture “crash mode” memory dump to determine what causes the process to crash.
Typically in a hang or a memory leaks scenario, you take multiple dumps in few minutes interval. This helps to understand and analyze the state of the application over a period of time. One thing you need to keep in mind that a dump file contains only the state of the process at the time the dump was taken, so multiple dumps are useful when we want to see the process state over a time span.
Live debugging should be your first option since dump is limited to show conditions only of what already happened. Also you can’t set conditions in dump as you can do in live debugging. You capture dump when application is running in an environment live debugging is not at all possible. Here is a list of typical scenarios when we want to capture a dump file:
· Application hangs with high or low CPU cycle
· Application is leaking memory
· Application is crashing or throwing exceptions
· Heap corruption
The ADPlus.vbs script itself does not require any specific configuration to use it, however you might be prompted to change your default script interpreter from WScript.exe to CScript.exe. Allowing ADPlus.vbs to set CScript as the default script interpreter is strongly recommended. If you continue to use WScript as your default script engine, then you will see a separate window when you run a command against ADPlus.vbs. Optionally, you can place CScript at the beginning of your command line, but you must include the ‘.vbs’ extension on ADPlus.vbs.
AdPlus creates dump files. These dumps contain stack information about the process(s) you are monitoring. This dump file and stack information relies on symbols (.pdb) to resolve that stack information. If a symbol path is not specified in the command line, it will use the _NT_SYMBOL_PATH environment variable if one is set, otherwise stack information in the dump will be limited. Here is how you can set the symbol path at the machine level.
To use AdPlus, you must specify a series of command-line options or arguments to the script. At a minimum, AdPlus requires two command-line options: one that specifies the mode of operation, and one that specifies a target process to operate against.
C:> Adplus.vbs –hang –p 123 –o C:Dumps
First, a run mode is required when passing arguments; here we used –hang. The run mode has two options –hang and –crash. The second mandatory parameter is the process to monitor. Here we are using the –p switch to tell ADPlus.vbs to capture a hang dump of a process with the process ID of 123. When AdPlus is running in hang mode, AdPlus must be started after the process hangs or is consuming high CPU utilization.
C:> Adplus.vbs –crash –p 123 –o C:Dumps
Crash mode is supported in a Terminal Server session on Windows XP and later versions of Windows. Hang mode will work inside a Terminal Server session on any platform. When AdPlus is running in crash mode, AdPlus must be started before the process crashes or becomes unstable.
When ADPlus.vbs is running in crash mode, a debugger remains attached to each process that is specified on the command line for the lifetime of that process. To manually detach the debugger from the process, you must maximize the debugger window and press CTRL+C to break into the debugger.
WinDbg is the ultimate debugger both for User mode and Kernel mode debugging. WinDbg is used mostly for analyzing dumps, yet you can capture dumps with this just like AdPlus.
In our next article on this series, we will see step-by-step examples of capturing dumps with AdPlus and then analyzing the dump with WinDbg.
In future posts, we will deal with this vast subject with one scenario at a time including but not limited to memory leak, High & low CPU hang, Heap corruption, Race condition and crash/exception both managed and unmanaged environments. We will also see how to leverage WinDbg with extension DLLs to make our debugging more pleasant.
HOWTO: Use AdPlus to troubleshoot "hangs" and "crashes" http://support.microsoft.com/default.aspx?scid=kb;en-us;Q286350
Posted By: Enamul Khaleque