Out of Pool Paged memory on 32-bit Windows Server 2003
Lately, I have been assisting customers who are still using 32-bit Windows Server 2003 and inevitably running out of kernel pool memory. When one of the kernel pools (Pool Paged and Pool Nonpaged) are full (meaning a memory allocation to one of these pools fails due to a lack of free space), then applications or even the system might hang or malfunction, so this is a serious consideration. In any case, the primary problem here is 32-bits.
The largest number that can be represented by 32-bit is 4 GB. That 4 GB is divided into user mode (process memory) and kernel mode. By default, this gives the kernel 2 GB to work with. The kernel memory is then divided further into a Page Frame Number (PFN) database, Pool Paged, Pool Nonpaged, and the rest of the free memory is given to Free System Page Table Entries (PTEs). In Windows Server 2003, Pool Paged and Pool Nonpaged have a specific size. This means that the system can run out of either pool and hang when there is still plenty of free kernel virtual memory elsewhere. This is why 32-bit versions of Windows Vista and Windows Server 2008 and later allow the full kernel virtual address space to be used.
Instead of me going into all of the complexities of how to detect this problem and how to troubleshoot it, I will just go into these recommendations if you suspect the system it out of kernel pool memory.
- Migrate to 64-bit Windows Server: 64-bit has 8 TB of virtual address space for kernel memory which will effectively solve this issue, but I realize that it might be difficult to go to 64-bit Windows Server. All drivers on a 64-bit version of Windows Server must be 64-bit, but 32-bit applications can run on 64-bit Windows Server.
- Upgrade to 32-bit Windows Server 2008: If it is impossible to go to a 64-bit version of Windows Server, then at the very least consider upgrading to 32-bit Windows Server 2008. This version of Windows Server allow Pool Paged and Pool NonPaged to expand to up to just less than 2 GB assuming there is more than 2 GB of physical memory installed – Pool NonPaged’s maximum is 75% of physical memory (RAM).
- Disable Dynamic Memory (hot-add): If the physical system has the capability to add more physical memory (RAM) to the system, then the Page Frame Number (PFN) database in Windows is likely larger than needed. To reduce the PFN database to a smaller size, set the DynamicMemory registry key to 1 which means 1 GB. The system realizes it is too small and resets it to the actual memory installed. This will likely increase the size of Pool Paged Bytes by 100 MB which could make the Pool Paged size to 169 MB.
How to Configure the Paged Address Pool and System Page Table Entry Memory Areas
The DynamicMemory registry key is located under the following registry key:
HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
- View the actual amount of Pool Paged maximum size: Use Process Explorer with the Debugging Tools for Windows (point Process Explorer to the dbghelp.dll file that comes with the Debugging Tools for Windows) and a symbol path of “SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols” (internet access required) to see the maximum size of the kernel pool paged.
- Look for kernel insufficient memory events: In the System event logs, look for Event ID 2019 and 2020 messages. If the system has these events, then it indicates that the system ran out of kernel virtual memory.
- Consider adjusting PoolPagedSize with CAUTION: Be *very* careful with this!!! The PoolPagedSize registry key (see the link for more information) can be set to a larger amount than what it is, *but* this will take away from the System PTEs and your system is borderline for running out of those as well. Only do this if a Microsoft Support Professional has recommended this.
- Use Poolmon.exe: If the pool paged memory usage is too much, then use Poolmon.exe to identify which drivers are consuming the most. The download and usage of Poolmon is described in this blog entry by Mark Russinovich (author of the Windows Internals book and creator of the Sysinternals tools).
Mark Russinovich’s blog entry on Windows pool memory troubleshooting: