question

TZacks-2728 avatar image
0 Votes"
TZacks-2728 asked Viorel-1 commented

How to mention List<T> size before two element join

See how my code. where i am joining two datatable and stored the result set into List<T>

 List<QCHelper> BogeyConfigList = null;
 BogeyConfigList = (from bogyconfiglist in Bogeylist.AsEnumerable().AsParallel() 
 join LiList in list.AsEnumerable().AsParallel()             
 on new
 {
     val = bogyconfiglist.LineItem.Trim().ToUpper(),
     val1 = bogyconfiglist.Section.Trim().ToUpper()
 }
 equals new
 {
     val = LiList.LI.Trim().ToUpper(),
     val1 = LiList.Section.Trim().ToUpper()
 }
 into conbogylist
 from confg in conbogylist.DefaultIfEmpty()
 select new QCHelper()
 {
     Section = bogyconfiglist.Section,
     Li = bogyconfiglist.LineItem,
     CrossCalc1Q = confg == null ? string.Empty : (confg.CrossCalc1Q == null ? "" : confg.CrossCalc1Q.Replace("~9999", string.Empty).Trim()),
     CrossCalc2Q = confg == null ? string.Empty : (confg.CrossCalc2Q == null ? "" : confg.CrossCalc2Q.Replace("~9999", string.Empty).Trim()),
     CrossCalc3Q = confg == null ? string.Empty : (confg.CrossCalc3Q == null ? "" : confg.CrossCalc3Q.Replace("~9999", string.Empty).Trim()),
     CrossCalc4Q = confg == null ? string.Empty : (confg.CrossCalc4Q == null ? "" : confg.CrossCalc4Q.Replace("~9999", string.Empty).Trim()),
     CrossCalcFY = confg == null ? string.Empty : (confg.CrossCalcFY == null ? "" : confg.CrossCalcFY.Replace("~9999", string.Empty).Trim()),
     AllowComma = confg == null ? false : confg.AllowComma,
     AllowedDecimalPlace = confg == null ? string.Empty : confg.AllowedDecimalPlace,
     AllowPercentageSign = confg == null ? false : confg.AllowPercentageSign,
     CurrencySign = confg == null ? string.Empty : confg.CurrencySign,
     IsQcCheck = confg == null ? false : confg.QCCheck,
     QcType = confg == null ? string.Empty : confg.QCType,
     FormulaLiConfig = confg == null ? string.Empty : (confg.StandrdFormula == null ? "" : confg.StandrdFormula.Replace("~9999", string.Empty).Trim()),
     xFundCode = bogyconfiglist.xFundCode == null ? string.Empty : bogyconfiglist.xFundCode
 }).Distinct().ToList<QCHelper>();

resultset stored into BogeyConfigList but i heard that List will have many empty items in it when data will be stored there and those empty item will consume memory.

so before join how could i declare List<T> with proper size as a result join result set can accommodate into List<T> ?

Please guide me how to handle this situation. i have like series of many joins in my routine where i am not able to mention List<T> size because i do not in advance how many data will be there after join in result set. discuss best approach to handle my scenario where i can save some memory. thanks

dotnet-csharp
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Viorel-1 avatar image
0 Votes"
Viorel-1 answered Viorel-1 commented

The list is created by ToList based on available elements, which are returned by Distinct, therefore no huge extra allocation is expected, which sometimes appears in case of other operations with lists.

In order to minimise the unused memory, you can execute BogeyConfigList.TrimExcess() at the end.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

repeated calling of TrimExcess() after many LINQ join hit the performance ? please advise. thanks

0 Votes 0 ·

Since this is an additional operation, it requires additional processing effort, which maybe is negligible. It can be evaluated experimentally using various data.

If you do not experience memory insufficiency, I think that you are not obliged to economise the memory obsessively.

0 Votes 0 ·
karenpayneoregon avatar image
1 Vote"
karenpayneoregon answered karenpayneoregon edited

This is an extremely simple example done on string but can be any type done here in a unit test method.

Example 1


 [TestMethod]
 public void LimitListToSize()
 {
     int maxElements = 2;
     var list = new List<string>();
     list.Capacity = maxElements;
    
     while (list.Count <= maxElements -1)
     {
         list.Add("X");
     }
    
     Assert.AreEqual(list.Count, maxElements);
        
 }


Example 2


Uses the following class. And the following container

 public class Person
 {
     public int Id { get; set; }
     public string FirstName { get; set; }
     public string LastName { get; set; }
 }

Unit test

 [TestMethod]
 public void LimitedPeople()
 {
     // arrange
     int maxItems = 3;
        
     var people = new List<Person>
     {
         new Person() {Id = 1, FirstName = "Karen", LastName = "Payne"},
         new Person() {Id = 2, FirstName = "Jim", LastName = "Gallagher"},
         new Person() {Id = 3, FirstName = "Anne", LastName = "Smith"},
         new Person() {Id = 4, FirstName = "Bob", LastName = "Lebow"},
         new Person() {Id = 5, FirstName = "Zack", LastName = "Anderson"}
     };
    
     // act
     LimitedList<Person> limitedList = new LimitedList<Person>(maxItems);
     foreach (var person in people)
     {
         limitedList.Add(person);
     }
    
        
     List<Person> results = limitedList.ToList();
        
     // assert
     Assert.IsTrue(results.Count == maxItems);
 }


85889-figure1.png



figure1.png (2.5 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TimonYang-MSFT avatar image
0 Votes"
TimonYang-MSFT answered TZacks-2728 commented

declare List<T> with proper size

I'm afraid this won't work.

Although List is using arrays as backing storage, the expansion mechanism of List causes its size to change according to actual conditions.

EnsureCapacity method in the source code of List:

         // Ensures that the capacity of this list is at least the given minimum
         // value. If the currect capacity of the list is less than min, the
         // capacity is increased to twice the current capacity or to min,
         // whichever is larger.
         private void EnsureCapacity(int min) {
             if (_items.Length < min) {
                 int newCapacity = _items.Length == 0? _defaultCapacity : _items.Length * 2;
                 // Allow the list to grow to maximum possible capacity (~2G elements) before encountering overflow.
                 // Note that this check works even when _items.Length overflowed thanks to the (uint) cast
                 if ((uint)newCapacity > Array.MaxArrayLength) newCapacity = Array.MaxArrayLength;
                 if (newCapacity < min) newCapacity = min;
                 Capacity = newCapacity;
             }
         }

If you are only worried about possible empty items, you may consider processing the empty values after getting the results, like this:

  var re = strs.Where(s => !string.IsNullOrWhiteSpace(s)).ToList();

A post for your reference: how to remove empty strings from list, then remove duplicate values from a list


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Misunderstanding occurred.

when we add item to list then its capacity increase for which more memory allocation occurred for list. so if my join return 10 result and 10 result will be stored in BogeyConfigList but memory will be allocated more than 10 items. so i like to know how could i mention fix size for my list when join data will be stored ?

i can use TrimExcess() function to remove unused items from list which save some memory but the problem is repeatedly calling TrimExcess() after each LINQ join may hit the performance.

so looking for best guide line. thanks

0 Votes 0 ·