ASP.NET Memory Investigation
This is a bit of a continuation of ASP.NET Memory Issue: High memory usage in a 64bit w3wp.exe process so if you haven't checked it out you might want to just glance over it before reading this one to get the context of the problem and some notes on 64 bit debugging.
Before I go into the details I just want to mention that what I will talk about does not only apply to 64 bit debugging even though I am using a 64 bit dump, you can just as easily see the problems I will talk, about and use the same techniques to discover them in a 32 bit process/dump.
In my previous posts the dumps have normally been pretty clean, and a lot of times they have only been suffering from one single memory issue, to make it easier to illustrate how a small detail can cause a big havoc. As we know though, the world is seldom or never black and white and in many cases there are more than one reason for high memory usage in a process.
This particular dump has various high memory consumers (apart from the issue described in the previous post) so it gives me an opportunity to show a few different common high memory culprits.
In order to not give out any proprietary information I have changed the contents of strings and names of certain custom classes in this dump, so if you are a nitpicker and find inconsistencies in string lengths or similar, that is why:)
Debugging details of the memory investigation
I am going to concentrate on the .NET GC heaps in this case, and freely ignore any objects that are not rooted, and for now attribute them to the issue described in the previous post.
The GC Heaps account for 1,502,201,224 bytes out of the 1,6 GB used for the dump, and out of this approximately 800 MB-1GB is Gen 0 objects (see the previous post), so the interesting thing is to figure out where the rest is going, I.e. the memory that is really sticking around.
Looking at the bottommost ~20 lines of !dumpheap -stat, showing the individual objects that used up the most memory, we get...
0:000> !dumpheap -stat ... 0x0000064253bda8f8 23,805 5,713,200 System.Xml.Schema.XmlSchemaElement 0x00000642bcf83908 126,788 6,085,824 System.Web.UI.WebControls.ListItem 0x00000642bcee8240 53,173 6,380,760 System.Web.UI.LiteralControl 0x00000642b77fcfb0 13,351 6,835,712 System.Data.DataTable 0x00000642bcee9a28 245,535 7,857,120 System.Web.UI.Pair 0x00000642bceabd08 263,432 8,429,824 System.Web.UI.StateBag 0x00000642788e5a88 172,973 9,714,456 System.Int32 0x00000642788405d8 124,338 10,941,744 System.Collections.Hashtable 0x00000642b7800d20 249,035 11,953,680 System.Data.DataRowView 0x0000064274e57ce8 300,268 12,010,720 System.Collections.Specialized.HybridDictionary 0x00000642bcea8920 137,011 12,056,968 System.Web.UI.Control+OccasionalFields 0x0000064274e57e70 278,134 13,350,432 System.Collections.Specialized.ListDictionary 0x0000064278827060 566,228 13,589,472 System.Int32 0x0000064253bcd750 466,140 14,916,480 System.Xml.Schema.BitSet 0x00000642788bd790 466,141 14,916,544 System.UInt32 0x00000642bceca168 514,257 16,456,224 System.Web.UI.StateItem 0x0000064253bcd090 444,728 17,789,120 System.Xml.XmlQualifiedName 0x000006427883e400 457,745 18,309,800 System.Collections.ArrayList 0x00000642bceb4f98 137,019 20,826,888 System.Web.UI.WebControls.TableCell 0x0000064274e57fe0 559,345 22,373,800 System.Collections.Specialized.ListDictionary+DictionaryNode 0x0000064253bc1c80 571,126 22,845,040 System.Xml.NameTable+Entry 0x00000642b78659d0 14,096 24,055,168 System.Data.RBTree`1+Node[[System.Int32, mscorlib]] 0x0000064278824860 784,204 25,094,528 System.Decimal 0x000006427883f770 565,270 27,132,960 System.Collections.ArrayList+ArrayListEnumeratorSimple 0x00000642b77fe2f0 124,259 28,828,088 System.Data.DataColumn 0x00000642788dbf50 30,668 29,910,520 System.Byte 0x00000642b7820d60 19,193 34,319,192 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]] 0x00000642b77fe7c0 377,440 36,234,240 System.Data.DataRow 0x00000642788db830 71,095 38,538,856 System.Char 0x00000642788fe918 135,194 50,900,832 System.Collections.Hashtable+bucket 0x00000642788e3b70 48,047 138,801,256 System.Decimal 0x00000642788d4758 897,213 172,527,160 System.Object 0x000006427881aaf8 3,013,208 197,958,896 System.String 0x0000000000146e90 12,107 220,141,008 Free Total 15,904,890 objects, Total size: 1,502,201,224 Fragmented blocks larger than 0.5 MB: Addr Size Followed by 00000000934858a0 25.3MB 0000000094dc8d50 System.Threading._ThreadPoolWaitCallback 0000000094dc8dc0 11.4MB 000000009592b4e0 System.Threading._ThreadPoolWaitCallback 000000009592b550 2.3MB 0000000095b6be48 System.Threading._ThreadPoolWaitCallback 0000000095b6beb8 2.1MB 0000000095d7a828 System.Threading._ThreadPoolWaitCallback 0000000095d7a898 0.7MB 0000000095e350b8 System.Threading._ThreadPoolWaitCallback 0000000095e35128 2.7MB 00000000960e7968 System.Threading._ThreadPoolWaitCallback ...
I have grouped them together by color to get a bit of an overview of what we are looking at... and categorized them like this
- XML Related Items
- Data Related Items
- UI + Web Controls Related Items
- State and Cache related items
In some cases, like in the case of Data, i have added UInt32 for example because I know from experience that a lot of these arrays will be arrays of UInt32 objects in datasets. The same goes for HashTables and ListDictionaries in Cache and Session State. Of course not all of these will be directly related to Data or Cache but for rough categorization it works well to put them in those buckets.
Byte, Object, Char and String are all types of objects that you will always see at the bottom of the !dumpheap -stat output since they are used everywhere, and you can read some of my previous memory investigation posts to get more details about this.
64-bit specific memory information
Since pointers are quad-word rather than DWORDs in 64-bit processes, a lot of the objects here, such as the Object and some of the collections (excluding strings and Byte for example) are double the size they would be in a 32-bit process.
A lot of Free objects (objects that are marked collected, but the space has not been reused or compacted) usually means one of two things.
1. The garbage collector hasn't compacted in a while (usually the case on the workstation build)
2. You have a lot of pinned object disabling the GC from compacting the heaps
In this case, the reason is #1, but not because we are running the workstation build, but because we are running 64-bit framework RTM and have very infrequent collections. See the previous post for a lengthier discussion and resolution.
Xml Related Items
When we have a lot of Xml Related items in the bottom part of !dumpheap -stat I usually look for a more top-level object type like XmlDocument since all other Xml related objects usually tend to be members or members of members of XmlDocuments. In other words, if you were to get rid of the XmlDocument, the other ones would go as well. The reason I look for a top-level object is because there are usually fewer of them so it is easier to work with them in order to find out why they stick around.
0x0000064253bc0c10 5,048 1,534,592 System.Xml.XmlDocument
And then i dump out a few and do !gcroot on them to find out why they are rooted...
0:000> !dumpheap -mt 0x0000064253bc0c10 Using our cache to search the heap. Address MT Size Gen 0x00000000800dfb98 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800e3578 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800e6f58 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800ea938 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800ee318 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800f1cf8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800f56d8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800f90b8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800fb8a0 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x00000000800fe5a8 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x0000000080102c28 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x0000000080106f58 0x0000064253bc0c10 304 2 System.Xml.XmlDocument 0x000000008010a938 0x0000064253bc0c10 304 2 System.Xml.XmlDocument ... 0:000> !gcroot 00000000800dfb98 ... DOMAIN(000000000017E210):HANDLE(Strong):4b41308:Root: 00000000c0020990(System.Threading._TimerCallback)-> 00000000c0020908(System.Threading.TimerCallback)-> 00000000c001b530(System.Web.Caching.CacheExpires)-> 00000000c001b568(System.Object)-> 00000000c001bef8(System.Web.Caching.ExpiresBucket)-> 00000001008fbf40(System.Web.Caching.ExpiresPage)-> 00000001008fbff8(System.Web.Caching.ExpiresEntry)-> 000000010001ebd8(System.Web.Caching.CacheEntry)-> 000000010001eb98(System.Web.SessionState.InProcSessionState)-> 00000000800dd340(System.Web.SessionState.SessionStateItemCollection)-> 00000000800dd3f0(System.Collections.Hashtable)-> 00000000802fe7e0(System.Collections.Hashtable+bucket)-> 00000000800e09e8(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)-> 00000000800dfb98(System.Xml.XmlDocument) ...
...so most of them are rooted in an InProcSessionState object which in turn is rooted in cache. (This is always the case for inproc session objects). So in other words, these XmlDocuments are stored in Session scope and more specifically in the session variable called Classes.
0:000> !do 00000000800e09e8 Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry MethodTable: 0000064274e6c8c8 EEClass: 0000064274efd388 Size: 32(0x20) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_MSIL\System\126.96.36.199__b77a5c561934e089\System.dll) Fields: MT Field Offset Type VT Attr Value Name 000006427881aaf8 400116d 8 System.String 0 instance 00000001400dd3a8 Key 0000064278818fb0 400116e 10 System.Object 0 instance 00000000800dfb98 Value 0:000> !do 00000001400dd3a8 Name: System.String MethodTable: 000006427881aaf8 EEClass: 000006427892f7d8 Size: 46(0x2e) bytes GC Generation: 2 (C:\WINDOWS\assembly\GAC_64\mscorlib\188.8.131.52__b77a5c561934e089\mscorlib.dll) String: Classes Fields: MT Field Offset Type VT Attr Value Name 0000064278827060 4000096 8 System.Int32 1 instance 11 m_arrayLength 0000064278827060 4000097 c System.Int32 1 instance 10 m_stringLength 00000642788216d8 4000098 10 System.Char 1 instance 50 m_firstChar 000006427881aaf8 4000099 20 System.String 0 shared static Empty >> Domain:Value 0000000000120a60:00000642787c36e0 000000000017e210:00000642787c36e0 << 00000642788db830 400009a 28 System.Char 0 shared static WhitespaceChars >> Domain:Value 0000000000120a60:00000000bfff0550 000000000017e210:00000000bfff7390 <<
If you want to dig deeper into this you may be interested in the following two posts:
A few of the Xml related items such as some of the XmlQualifiedName objects for example actually turn out to be rooted in DataSets but I just want to do a pretty quick and dirty memory investigation here to get a good overview of what is going on.
Data Related Items
For the data related items i usually use System.Data.DataSet as a top-level item, and if I would do that in this case (following the same procedure as for XmlDocument) I would find that some of them are stored in session scope and some in cache directly.
State Related Items
If we have a lot of state related items that could be an indication of a couple of things
- Lots of caching
- Lots of in-proc session state (stored in cache)
- Lots of aspx or ascx pages around with related viewstate (System.Web.UI.Pair, System.Web.UI.StateBag)
I am going to leave #3 for a bit and discuss the other two, since we already know that we have a lot of DataSets and XmlDocuments in Session and cache.
If we take a look at the cache and it's size we find that it is ~550 MB...
0x00000642bcece978 1 24 System.Web.Caching.Cache 0:000> !dumpheap -mt 0x00000642bcece978 Address MT Size Gen 0x00000000c001b288 0x00000642bcece978 24 2 System.Web.Caching.Cache ... 0:000> !objsize 0x00000000c001b288 sizeof(00000000c001b288) = 557,714,296 ( 0x213e0b78) bytes (System.Web.Caching.Cache)
This is a pretty big chunk of the memory so I think it is safe to say that other than the issue described in the previous post, this is where we should focus our efforts if we want to reduce the amount of memory used.
If we then go one step further to try to see how much this is session objects we can dump out the InProcSessionState objects and check the size of one of those...
0x00000642bcedf988 601 38,464 System.Web.SessionState.InProcSessionState 0:000> !dumpheap -mt 0x00000642bcedf988 Address MT Size Gen 0x00000000800d9ab8 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState 0x00000000800dba40 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState 0x00000000800dbb58 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState 0x00000000800dbcc0 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState 0x00000000800dbdd8 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState 0x00000000800dbef0 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState 0x00000000800dc008 0x00000642bcedf988 64 2 System.Web.SessionState.InProcSessionState ... 0:000> !objsize 0x00000000800dc008 sizeof(0x00000000800dc008) = 557,714,296 ( 0x213e0b78) bytes (System.Web.SessionState.InProcSessionState)
Ok, so we have 601 active sessions and the size of just one of those sessions is the same amount as the cache itself which just doesn't make sense, and in fact offline I had taken !objsize of a sample of the datasets and XmlDocuments and although it is a sizeable chunk they will not rack up to 550 MB together, so something is fishy.
What is actually happening is that we are storing something in session scope that eventually has a link to a HttpContext, which has a link to cache, causing the actual size of the cache to be included in !objsize for the individual session objects.
When you see something like this, you should pay attention, because although the "circular reference" will be broken by taking the object out of session state, the problem is that this normally happens when you have added something to session state which has a link to an aspx page or ascx page or similar. Very much like in the case of the EventHandlers issue or in the CacheItemRemovedCallback, but as you will soon see, this is slightly different.
UI + Web Controls Related Items
The final stage of this memory investigation is to look at the UI and Web Controls related items. We already know that our memory can roughly be divided into memory usage caused by the issue in this post, and memory used for session state. However what we don't know is causing us to have so much session state since the data and xml related items don't seem to match the total size.
If we pick the TableCells and work out who is using them, just like in the data and xml cases, we get...
0x00000642bceb4f98 137,019 20,826,888 System.Web.UI.WebControls.TableCell 0:000> !dumpheap -mt 0x00000642bceb4f98 Using our cache to search the heap. Address MT Size Gen 0x00000000806321b0 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080632850 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080632a38 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080632b90 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080632ce8 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080632e40 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080632f98 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x00000000806331b0 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080633308 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080633460 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x00000000806335b8 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0x0000000080633710 0x00000642bceb4f98 152 2 System.Web.UI.WebControls.TableCell 0:000> !gcroot 0000000080632850 ... DOMAIN(000000000017E210):HANDLE(WeakSh):4b412f8:Root: 00000000c0020b80(System.Threading._TimerCallback)-> 00000000c0020af8(System.Threading.TimerCallback)-> 00000000c001df60(System.Web.Caching.CacheExpires)-> 00000000c001df98(System.Object)-> 00000000c001e928(System.Web.Caching.ExpiresBucket)-> 00000000c0adbf78(System.Web.Caching.ExpiresPage)-> 00000000c0adc030(System.Web.Caching.ExpiresEntry)-> 0000000140122e48(System.Web.Caching.CacheEntry)-> 0000000140122e08(System.Web.SessionState.InProcSessionState)-> 00000000c01632a0(System.Web.SessionState.SessionStateItemCollection)-> 00000000c0163350(System.Collections.Hashtable)-> 00000001002feb20(System.Collections.Hashtable+bucket)-> 00000000c0403398(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)-> 000000008062b008(MyControls.MyCustomGridControl)-> 0000000080611c68(System.Web.UI.WebControls.DataGrid)-> 0000000080611da0(System.Web.UI.Control+OccasionalFields)-> 0000000080629c20(System.Web.UI.ControlCollection)-> 0000000080632098(System.Object)-> 0000000080631fe8(System.Web.UI.WebControls.ChildTable)-> 0000000080632660(System.Web.UI.Control+OccasionalFields)-> 00000000806326b8(System.Web.UI.WebControls.Table+RowControlCollection)-> 00000000806326f0(System.Object)-> 0000000080632780(System.Web.UI.WebControls.DataGridItem)-> 00000000806329a8(System.Web.UI.Control+OccasionalFields)-> 0000000080632a00(System.Web.UI.WebControls.TableRow+CellControlCollection)-> 00000000806330f0(System.Object)-> 0000000080632850(System.Web.UI.WebControls.TableCell)
So it appears that for some reason we are storing a web control (MyControls.MyCustomGridControl) in session state...
This may not look like a very bad idea, in fact I can completely see why this is done, since it is a fast and easy way to store the state of the data that this particular user was working with and use this throughout the session, but there are some major issues with this...
All the data related to the control will now be linked to session and can't be garbage collected before this object goes out of session, while what we really wanted was to just store the actual data and perhaps the rowfilters. And among this data is not only UI related items, like WebControls.DataGridItem etc. which will be relatively useless to us once the page that held the grid has finished executing... but there is also a link to the page it was loaded on in multiple ways.
1. In the parent field of the control
2. As the target for many of the events related to the grid (like in the EventHandlers memory investigation)
And since we have a link to the page, we also keep everything that belongs to it in session state, such as its viewstate, its other controls etc. which can definitely be a whole lot of data, and this data can not be collected until the session expires.
This also explain why we have a circular reference with the cache going on, since the page has a context which links to the cache...
So what should you do instead of storing controls in session scope? The best would probably be to store the data you need in a separate object and store that in session scope if it takes too long time or is otherwise impractical to gather when needed.
When it comes to memory investigations where most of the memory is .NET GC memory, the memory investigation is often a paint-by-numbers process where the steps go
- Run !dumpheap -stat and look for items that stick out, or try to group/categorize the bottommost items in the list to figure out who are your main memory hoggers
- !gcroot the objects to figure out why they are still around... (Note! Make sure you !gcroot a couple of them before making any big decisions based on the results)
- Rinse and repeat until you have accounted for most of the memory.
The tricky part is of couse figuring out what sticks out, grouping them properly, and understanding the results of !gcroot:) but hopefully this blog helps you with some of that.
Until next time,