.NET Memory Leak reader email: Are you really “leaking” .net memory
I get several emails every week through the blog asking for help on various issues. Unfortunately due to time constraints I can’t really look at them all on an individual bases but I thought this particular request might be interesting to share.
The problem description was “One of our w3wp.exe’s just keep on growing (slowly), and never releasing memory back to the pot. The main offender appears to be System.String and System.Object[]” and the email then had some !dumpheap –stat output (from windbg and sos)
0:000> .load sos
0:000> !dumpheap -stat
...
654060a8 5622 292344 System.Data.SqlClient.SqlBuffer
79109778 5505 308280 System.Reflection.RuntimeMethodInfo
79131d04 1598 329732 System.Boolean[]
65408b8c 5568 356352 System.Data.DataRow
654359c8 160 395136 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
65412bb4 160 395136 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
6540b178 5664 407808 System.Data.DataColumnPropertyDescriptor
65405fdc 5741 734848 System.Data.SqlClient._SqlMetaData
654088b4 5664 838272 System.Data.DataColumn
7a75a878 55225 883600 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
7912dd40 709 891172 System.Char[]
79102290 100163 1201956 System.Int32
7910be50 116208 1394496 System.Boolean
7912d9bc 1685 2078376 System.Collections.Hashtable+bucket[]
7912d7c0 13131 2205228 System.Int32[]
79131840 1402 2206136 System.DateTime[]
79131b20 793 2485548 System.Decimal[]
7912dae8 1316 3573540 System.Byte[]
7912d8f8 24039 5152404 System.Object[]
790fd8c4 272294 21045644 System.String
Total 781834 objects
The reason I thought this request was interesting is because it brings up an important point about troubleshooting memory issues.
Typically when I work with a customer on a support case and they mention that they have memory issues I usually start off by asking:
How did you determine that you have a memory leak or high memory usage?
This might sound like a simple question, i.e. you determine that you have a memory leak because memory keeps growing, or you determine that you have high memory usage because the process is using a lot of memory. However, it is usually a lot more complex than that.
For some applications it is normal to use up 1 GB of memory and for others 200 MB is an extremely high number. To determine what is high or not for your application you need to establish some type of baseline of how the application works under normal circumstances.
Also when it comes to high memory usage and potential memory leaks it is important to know what memory counters you are looking at. Is it virtual bytes, private bytes, .net memory (#Bytes in all heaps) and for when and how long you are looking at these counters.
For example, when working with .net applications, virtual bytes will go up in chunks since we allocate GC heap segments in chunks. How big these chunks are depend on what framework version you are using and if it is running the server GC or workstation GC. Also, as I have mentioned several times in previous blog posts, garbage collections do not occur at time intervals but rather when you allocate memory and reach the limits set for different generations. Therefore it is normal for a process to not return memory if no allocations are made.
For some discussions on virtual byte vs. private bytes, and how the GC works, you can have a look at:
.NET Memory Usage – A restaurant analogy
How does the GC work
Generational GC – A post-it analogy
ASP.NET Memory investigation
32 vs 64 bit memory usage
Defining the “where” of a memory leak
In the email above it was never stated how much memory would grow, or what it would grow to, or even how much private bytes the process was using when the dump was taken. I realize that it’s hard to remember to write everything in an email, or know what is important informaiton, but without this data it is very hard to say anything at all about what caused a potential memory increase.
Looking at the dumpheap –stat output above we can see that the process has roughly 40 MB of .net memory (by just adding up the major memory consumers). This to me is not a very big number, but again, that completely depends on what the normal memory usage of the process is. I often find that people stare themselves blind on the .net memory usage, but I think that before doing that, you need to use some critical thinking and determine if the 40 MB is even worth looking at. If the process is using up 700 MB at this time for example, then the 40MB is just a drop in the ocean, and any statements made about it are irrelevant.
If the private bytes at this time is around 150 MB, then the question becomes, is that really a big number? And you need to consider whether you have taken a memory dump before it started “leaking”. If the 150 MB is a high number for you, you need to look at what is using up the other 110 MB and you will probably find that a lot of it is going to assemblies etc.
Ideally if you have a perceived memory leak, you should take two memory dumps and compare the growth of the .net heap between them to see if the growth in the process is due to growth in .net memory. Or an easier way of doing this would be to compare the #Bytes in all heaps and private bytes over time.
So the conclusion here is that the !dumpheap output can’t be used for analysis without the supporting data around why/how and when you saw the problem.
Another interesting fact that was left out of the email was whether this application had always had this issue or if this started happening recently. If it started after the application had been live for some time we need to look at what changed.
System.String and array objects
Finally, I would like to address the comment around System.String and System.Object[] being the major offenders. I wrote an article in the beginning of this blog around examining !dumpheap –stat output, because it is very common that people focus in on the string and object[] objects when reviewing memory dumps.
The reason for this is that arrays and strings will always end up on the bottom of the !dumpheap –stat output since !dumpheap –stat will list objects by the total size of such objects. This happens because a) pretty much everything uses strings and arrays so there will always be a large amount of strings and arrays in any dump and b) because of how the size is calculated.
If you take an object like a dataset, the dataset basically consists of a few different pointers to data tables, dataset name etc. all in all the space used for the dataset object is around 80 bytes (just the pointers). The actual datatables and its rows and columns etc. are not really a part of the dataset, they are just linked from the dataset. Therefore if you have 10 datasets they would take up 800 total bytes, and if you have a 100 it would be close to 8kb, independently of how many rows/columns, blobs etc. are in the columns and rows. (note: actual dataset size varies with framework versions)
The size of strings and array objects however are variable. The size of a string is the actual size of it’s contents, and the size of an array is the size of all the pointers to the elements. Therefore the total size of all strings and arrays will for the most part hugely surpass the size of anything else in the process and thus they will always end up at the bottom.
I would say, if you look at a memory leak, forget about the arrays and strings unless they end up on the LOH (!dumpheap –min 85000) in which case you might want to look at the ones that end up on the LOH separately.
In summary
If you are doing a memory leak investigation, first get facts about actual memory usage, baseline memory usage, how memory grows over time etc. to get a clear picture of if you are actually looking at a “leak” or normal behavior, and if you determine that it is in fact an issue in that the app is using a lot more .net memory than expected, look beyond the strings an arrays on the first pass through the data.
Have fun,
Tess