I'm trying to parse a very large csv file with FileHelpers (http://www.filehelpers.net/). The file is 1GB zipped and about 20GB unzipped.
string fileName = @"c:myfile.csv.gz";
using (var fileStream = File.OpenRead(fileName))
{
using (GZipStream gzipStream = new GZipStream(fileStream, CompressionMode.Decompress, false))
{
using (TextReader textReader = new StreamReader(gzipStream))
{
var engine = new FileHelperEngine<CSVItem>();
CSVItem[] items = engine.ReadStream(textReader);
}
}
}
FileHelpers then throws an OutOfMemoryException.
Test failed: Exception of type 'System.OutOfMemoryException' was
thrown. System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown. at
System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount) at
System.Text.StringBuilder.Append(Char value, Int32 repeatCount) at
System.Text.StringBuilder.Append(Char value) at
FileHelpers.StringHelper.ExtractQuotedString(LineInfo line, Char
quoteChar, Boolean allowMultiline) at
FileHelpers.DelimitedField.ExtractFieldString(LineInfo line) at
FileHelpers.FieldBase.ExtractValue(LineInfo line) at
FileHelpers.RecordInfo.StringToRecord(LineInfo line) at
FileHelpers.FileHelperEngine1.ReadStream(TextReader reader, Int32
maxRecords, DataTable dt) at
FileHelpers.FileHelperEngine
1.ReadStream(TextReader reader)
Is it possible to parse a file this big with FileHelpers? If not can anyone recommend an approach to parsing files this big? Thanks.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…