List size limitation in C#

Question

This could possibly appear to be a nasty thing to ask but why do we have so short limit of number of objects in a list.

i wrote following code to test list size in C#

    List<int> test = new List<int>();            
    long test1 = 0;
    try
    {
        while (true)
        {
            test.Add(1);
            test1++;
        }
    }
    catch (Exception ex)
    {
        MessageBox.Show(test1 + "   |   " + ex.Message);
    }

and the size of list could only be 134217728

isn't that unfair :( ??? what is alternate way if i want to add objects even beyond 'integer' limits (i mean number of objects > 2^32) ???

Answer 1

A List<int> is backed by an int[]. You will fail as soon as a larger backing array cannot be allocated - and bear in mind that:

• There's a 2GB per-object limit in the CLR even in 64 bits (EDIT: as of .NET 4.5, this can be avoided for the 64-bit CLR - see <gcAllowVeryLargeObjects>)
• The list will try to allocate a backing array which is larger than what it immediately requires, in order to accommodate later Add requests without reallocation.
• During the reallocation, there has to be enough total memory for both the old and the new arrays.

Setting the Capacity to a value which will put the backing array near the theoretical limit may get you a higher cutoff point than the natural growth, but that limit will certainly come.

I would expect a limit of around 2²⁹ elements (536,870,912) - I'm slightly surprised you haven't managed to get beyond 134,217,728. How much memory do you actually have? What version of .NET are you using, and on what architecture? (It's possible that the per-object limit is 1GB for a 32-bit CLR, I can't remember for sure.)

Note that even if the per-object limit wasn't a problem, as soon as you got above 2³¹ elements you'd have problems addressing those elements directly with List<T>, as the indexer takes an int value.

Basically, if you want a collection with more than int.MaxValue elements, you'll need to write your own, probably using multiple backing arrays. You might want to explicitly prohibit removals and arbitrary insertions :)