.net - Processing large text file in C#

Question

I have 4GB+ text files (csv format) and I want to process this file using linq in c#.

I run complex linq query after load csv and convert to class?

but file size is 4gb although application memory double size of file.

how can i process (linq and new result) large files?

Thanks

Answer 1

Instead of loading whole file into memory, you could read and process the file line-by-line.

using (var streamReader = new StreamReader(fileName))
{
    string line;
    while ((line = streamReader.ReadLine()) != null)
    {
        // analize line here
        // throw it away if it does not match
    }
}

[EDIT]

If you need to run complex queries against the data in the file, the right thing to do is to load the data to database and let DBMS to take care of data retrieval and memory management.