{"id":829,"date":"2010-03-12T00:00:00","date_gmt":"2010-03-12T00:00:00","guid":{"rendered":"https:\/\/test.simple-talk.com\/uncategorized\/troubleshooting-nonpaged-and-paged-pool-errors-in-windows\/"},"modified":"2018-03-28T08:45:13","modified_gmt":"2018-03-28T08:45:13","slug":"troubleshooting-nonpaged-and-paged-pool-errors-in-windows","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/sysadmin\/general\/troubleshooting-nonpaged-and-paged-pool-errors-in-windows\/","title":{"rendered":"Troubleshooting Nonpaged and Paged Pool Errors in Windows"},"content":{"rendered":"<div id=\"PRETTY\">\n<p class=\"START\">I recently had an issue where, after a software change on our servers, we started to notice that some systems had become unstable and were regularly crashing.\u00a0 The crashes sometimes resulted in a blue-screen, but other times resulted in a machine which responded to ping, but little else, and had a completely unresponsive console.\u00a0 The only course of action was to power-cycle the crashed server; clearly, not a good thing to do when we&#8217;re dealing with production servers.<\/p>\n<p>Upon investigation, we found that immediately before the crash the servers would log event 2019 in the System log &#8211; &#8220;<em>The server was unable to allocate from the system nonpaged pool because the pool was empty<\/em>&#8220;.<\/p>\n<div class=\"ILLUSTRATION\">\n<p class=\"ILLUSTRATION\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/970-error.jpg\" alt=\"970-error.jpg\" width=\"432\" height=\"476\" \/><\/p>\n<\/div>\n<p class=\"CAPTION\">Figure 1 &#8211; Event 2019<\/p>\n<p>Thankfully, the error message in the event log gave us a clear indication as to <em>why<\/em> the systems were in trouble, and allowed us to troubleshoot and diagnose the problem.<\/p>\n<h2>About nonpaged pool<\/h2>\n<p>The nonpaged pool is memory which always resides in physical memory &#8211; it is never paged out.\u00a0 It is used by the kernel and also by device drivers installed on a system to store data which might be accessed in situations when page faults are not allowed.\u00a0 The amount of memory allocated to the nonpaged pool varies, and is determined as a function of operating system, processor architecture, and physical memory size. For example, 32-bit operating systems, with their smaller address spaces, have lower limits:<\/p>\n<ul>\n<li>32-bit Windows Server 2003 with 2GB or more of RAM will have a nonpaged pool limit of 256MB<\/li>\n<li>32-bit Windows Server 2008 will have a nonpaged pool limit of either 2GB or slightly more than 75% of physical memory, whichever is smaller<\/li>\n<\/ul>\n<p>64-bit operating systems, which have a much larger address space, have higher limits:<\/p>\n<ul>\n<li>64-bit Windows Server 2003 will have a nonpaged pool of either 128GB or 40% of physical memory, whichever is smaller<\/li>\n<li>64-bit Windows Server 2008 (or 2008 R2) will have a nonpaged pool limit of either 128GB or slightly more than 75% of physical memory, whichever is smaller<\/li>\n<\/ul>\n<p><em>Pool size data is from Mark Russinovich and David Solomon&#8217;s book &#8220;Windows Internals, 5th Edition&#8221;, and Mark Russinovich&#8217;s blog posting &#8220;<\/em><a href=\"http:\/\/blogs.technet.com\/markrussinovich\/archive\/2009\/03\/26\/3211216.aspx\"><em>Push the Limit&#8217;s of Windows: Paged and Nonpaged Pool<\/em><\/a><em>&#8220;.<\/em><\/p>\n<p>One way to see the nonpaged pool limit on a specific system is to install the <a href=\"https:\/\/docs.microsoft.com\/en-us\/windows-hardware\/drivers\/debugger\/debugger-download-tools\">Debugging Tools for Windows<\/a>, and then use Sysinternals&#8217; <a href=\"http:\/\/technet.microsoft.com\/en-us\/sysinternals\/bb896653.aspx\">Process Explorer<\/a> to display the pool size.\u00a0 (The debugging tools are required to provide access to debugging symbols.)<\/p>\n<p>Once the tools are downloaded and installed, launch Process Explorer and click <strong>Options -&gt; Symbol Configuration<\/strong>, point it to the <strong>dbghelp.dll<\/strong> file installed with the Debugging Tools, and configure <a href=\"http:\/\/support.microsoft.com\/kb\/311503\">Microsoft&#8217;s symbol server<\/a> as the symbol file path.<\/p>\n<div class=\"ILLUSTRATION\">\n<p class=\"ILLUSTRATION\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/970-symbols.jpg\" alt=\"970-symbols.jpg\" width=\"449\" height=\"246\" \/><\/p>\n<\/div>\n<p class=\"CAPTION\">Figure 2 &#8211; Process Explorer Symbol Configuration<\/p>\n<p>The nonpaged pool size can then be found on the System Information dialog (click <strong>View -&gt; System Information<\/strong> or press <strong>Ctrl+I<\/strong>):<\/p>\n<div class=\"ILLUSTRATION\">\n<p class=\"ILLUSTRATION\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/970-SysInfo.jpg\" alt=\"970-SysInfo.jpg\" width=\"599\" height=\"547\" \/><\/p>\n<\/div>\n<p class=\"CAPTION\">Figure 3 &#8211; Nonpaged pool allocation and limit on 32-bit Windows Server 2003 with 1GB RAM<\/p>\n<h2>Back to the problem<\/h2>\n<p>We were monitoring memory usage on one of the constantly crashing systems, including the performance counter for nonpaged pool allocation &#8211; <em>Memory\\Pool nonpaged bytes<\/em>.\u00a0 The orange line in Figure 4 is nonpaged pool usage, and the plot shows usage growing steadily over time, and then reducing sharply whenever the system is rebooted.<\/p>\n<div class=\"ILLUSTRATION\">\n<p class=\"ILLUSTRATION\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/970-MemUsage.jpg\" alt=\"970-MemUsage.jpg\" width=\"630\" height=\"246\" \/><\/p>\n<\/div>\n<p class=\"CAPTION\">Figure 4 &#8211; Memory use over time<\/p>\n<p>We quickly realised that what we were seeing was most likely a memory leak in a driver or kernel component.\u00a0 Armed with this knowledge and data, the next step was clearly to find out exactly <em>which<\/em> driver or component is consuming the pool.\u00a0<\/p>\n<p>The tool for this job is the Memory Pool Monitor, <strong>poolmon.exe<\/strong>, which is included in the Windows Support Tools on the Windows Server 2003 CD, or alternatively can be downloaded from the <a href=\"http:\/\/www.microsoft.com\/downloads\/details.aspx?familyid=96A35011-FD83-419D-939B-9A772EA2DF90\">Microsoft Download Centre<\/a> as part of the Windows Server 2003 Support Tools package. \u00a0Poolmon displays the amount of pool storage (both paged and nonpaged) in use, all of which is categorized by a pool tag, which is usually a four-character string used when calling the kernel APIs for allocating pool storage.<\/p>\n<p>After launching poolmon, press the &#8216;<strong>p<\/strong>&#8216; key to filter for paged or nonpaged pool, the &#8216;<strong>b<\/strong>&#8216; key to sort the output by bytes, or the &#8216;<strong>d<\/strong>&#8216; key to sort by the difference between pool allocations and pool frees. With the output set to nonpaged and sorted by bytes, the display could look similar to this:<\/p>\n<div class=\"ILLUSTRATION\">\n<p class=\"ILLUSTRATION\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/970-poolmon.jpg\" alt=\"970-poolmon.jpg\" width=\"600\" height=\"154\" \/><\/p>\n<\/div>\n<p class=\"CAPTION\">Figure 5 &#8211; Poolmon.exe<\/p>\n<p>The top line of the output is showing that the tag &#8220;<em>SbAp<\/em>&#8221; has made 2,187,628 allocations of 56 bytes and no frees, resulting in 122,507,168 bytes of nonpaged pool use &#8211; by far the biggest consumer on the system, and responsible for <em>over 60%<\/em> of the pool use.\u00a0 This looks like the likely cause of the memory leak.<\/p>\n<p>Now that we know the tag we&#8217;re looking for, we need to find out which device driver is using it, and there are a couple of ways to do this.\u00a0 If the tag is used by a kernel component or driver, and the Debugging Tools for Windows are installed, then the tag will be listed in the <strong>triage\\pooltag.txt<\/strong> file located in the debugging tools folder. If the tag isn&#8217;t listed in pooltag.txt, then we need to find it using the Sysinternals&#8217; <a href=\"http:\/\/technet.microsoft.com\/en-us\/sysinternals\/bb897439.aspx\">Strings<\/a> utility, <strong>strings.exe<\/strong>, to hunt it down.\u00a0 Since the tag is stored inside the driver file, and most driver files are in <strong>%SystemRoot%\\System32\\drivers<\/strong>, we can easily use strings.exe to quickly search all the files for the tag. So, the search for the &#8220;SbAp&#8221; tag returned one driver file: <strong>klif.sys<\/strong>.<\/p>\n<div class=\"ILLUSTRATION\">\n<p class=\"ILLUSTRATION\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/970-string.jpg\" alt=\"970-string.jpg\" width=\"628\" height=\"293\" \/><\/p>\n<\/div>\n<p class=\"CAPTION\">Figure 6 &#8211; Using strings.exe to find the driver<\/p>\n<p>Once we had identified the device driver, we could identify the manufacturer and get help from their technical support department.\u00a0 Thankfully, In this case we were able to contact the software vendor and get the problem solved very quickly, preventing any further crashes and loss of productivity.<\/p>\n<p>It&#8217;s worth bearing in mind that the same technique can also be used to troubleshoot paged pool problems as well, which will present as event ID 2020, with the text &#8220;<em>The server was unable to allocate from the system paged pool because the pool was empty<\/em>&#8220;. The only difference is to use poolmon to display the paged pool instead of nonpaged pool.<\/p>\n<p>The basic process in both cases is:<\/p>\n<ul>\n<li>Use the event log message to find out if you&#8217;re facing a paged or nonpaged pool problem<\/li>\n<li>Use <strong>poolmon.exe<\/strong> to find the offending tag<\/li>\n<li>Use <strong>pooltag.txt<\/strong> or <strong>strings.exe<\/strong> to identify the component or driver<\/li>\n<li>Enlist the vendor to fix the memory leak<\/li>\n<\/ul>\n<p>Whether you have a paged or nonpaged pool problem, once you have the right tools and know what to look for, these problems are really not especially difficult to troubleshoot.<\/p>\n<div class=\"note\">\n<p class=\"note\">This article was commissioned by Red Gate Software, engineers of ingeniously simple tools for optimizing your Exchange email environment. <br \/>\nLearn more about <a href=\"http:\/\/www.red-gate.com\/products\/Exchange\/index.htm?utm_source=simpletalk&amp;utm_medium=weblink&amp;utm_content=exchangenote&amp;utm_campaign=esa\">Exchange Server Archiver<\/a> and <a href=\"http:\/\/www.red-gate.com\/products\/PST_Importer\/index.htm?utm_source=simpletalk&amp;utm_medium=weblink&amp;utm_content=pstnote&amp;utm_campaign=PSTImporter\">PST Importer.<\/a><\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Ben Lye uncovered a memory leak in the nonpaged pool which was crashing his servers with disquieting regularity. Luckily it was relatively easy to troubleshoot, and he&#8217;s sharing the tools and techniques he used to get his servers back on track in double-quick time.&hellip;<\/p>\n","protected":false},"author":221845,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[32],"tags":[4887,4441,4871,5136],"coauthors":[48341],"class_list":["post-829","post","type-post","status-publish","format-standard","hentry","category-general","tag-general","tag-memory-leak","tag-sysadmin","tag-troubleshooting"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/829","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/221845"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=829"}],"version-history":[{"count":4,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/829\/revisions"}],"predecessor-version":[{"id":77777,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/829\/revisions\/77777"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=829"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=829"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=829"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=829"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}