Troubleshooting Pool Leaks Part 2 – Poolmon
In our previous article we discussed how to identify a pool leak using perfmon. Although it may be interesting to know that you have a pool leak, most customers are interested in identifying the cause of the leak so that it can be corrected. In this article we will begin the process of identifying what kernel mode driver is leaking pool, and possibly identify why.
Often when we are collecting data for a poor performance scenario there are two pieces of data that we collect. Perfmon log data is one, as we discussed in our previous article. The other piece of data is poolmon logs. The memory manager tracks pool usage according to the tag associated with the pool allocations, using a technique called pool tagging. Poolmon gathers this data and displays it in an easy to use format. Poolmon can also be configured to dump data to a log, and in some scenarios it is beneficial to schedule poolmon to periodically collect such logs. There are several available techniques to schedule poolmon, however that is beyond the scope of this article.
Poolmon has shipped with many different packages over the years; it is currently available with the Windows Driver Kit. If you install the WDK to the default folder, poolmon will be in “C:\Program Files (x86)\Windows Kits\8.0\Tools\x64”. Poolmon does not have dependencies on other modules in this folder; you can copy it to your other computers when you need to investigate pool usage.
How does pool tagging work? When a driver allocates pool it calls the ExAllocatePoolWithTag API. This API accepts a tag - a four-letter string - that will be used to label the allocation. It is up to a driver developer to choose this tag. Ideally each developer will choose a tag that is unique to their driver and use a different tag for each code path which calls ExAllocatePoolWithTag. Because each tag should be unique to each driver, if we can identify the tag whose usage corresponds with the leak we can then begin to identify the driver which is leaking the memory. The tag may also give the driver developer clues as to why the memory is being leaked, if they use a unique tag for each code path.
To view the pool usage associated with each tag run “poolmon -b” from a command prompt. This will sort by the number of bytes associated with each tag. If you are tracking pool usage over a period of time, you can log the data to a file with “poolmon -b -n poolmonlog1.txt”, replacing 1 with increasing numbers to obtain a series of logs. Once you have a series of logs you may be able to view usage increasing for a specific tag, in a corresponding fashion to what you see in perfmon.
When analyzing poolmon the important data is at the top. Typically the tag with the largest usage in bytes is the cause of the leak.
In the above data we can see that the tag with the most pool usage is “Leak”. Now that we know what tag is leaking we need to identify what driver is using this tag. Techniques for associating a leak with a tag vary, but findstr is often effective. Most drivers are located in c:\windows\system32\drivers, so that is a good starting point when looking for the driver. If you don’t find a result in that folder, go up a folder and try again, repeating until you get to the root of the drive.
C:\>findstr /s Leak *.sys
·∟ §£♂ Θ─☺ A╗☻ E☼"├Θ╡☺ Hï♣╔♂ ╞ $Θª☺ Hï♣:Hc┴ ┴ê\♦@ë
└δ_Aï ╞♣@∟ ☺ë♣▓← Aï@♦ë♣¼← δCAâ∙♦u╓AïAìI♦A╕Leak;┴☼B┴3╔ï╨ §
In the above output we can see that “Leak” is used in myfault.sys. If we hadn’t forced this leak with notmyfault, the next step in troubleshooting would be an internet search for the tag and the driver. Often such a search will allow you to identify a specific fault within the driver and a solution.
Don’t panic if findstr doesn’t find your tag, or if you find the tag but it is not unique to one driver. In future articles we will cover additional techniques for associating drivers with tags, and for associating allocations with specific code within a driver.