.net - OutOfMemoryException in Regex Matches when processing large files

Question

I've got an exception log from one of production code releases.

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.RegularExpressions.Match..ctor(Regex regex, Int32 capcount, String text, Int32 begpos, Int32 len, Int32 startpos)
   at System.Text.RegularExpressions.RegexRunner.InitMatch()
   at System.Text.RegularExpressions.RegexRunner.Scan(Regex regex, String text, Int32 textbeg, Int32 textend, Int32 textstart, Int32 prevlen, Boolean quick)
   at System.Text.RegularExpressions.Regex.Run(Boolean quick, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
   at System.Text.RegularExpressions.MatchCollection.GetMatch(Int32 i)
   at System.Text.RegularExpressions.MatchEnumerator.MoveNext()

The data it tries to process was about 800KB.

In my local tests it works perfectly fine. Have you ever seen similar behaviour, what can be the cause?

Shall I split the text before processing it, but obviously in that case regex might not match because the original file split from a random place.

My Regexes:

EDIT 2 :

I think this particular RegEx is causing the problem, when I test it out in an isolated environment it's eating the memory instantly.

((?:( |..|.|""|'|=)[/|?](?:[w#!:.?+=&@!$'~*,;/()[]-]|%[0-9a-f]{2})*)( |.|..|""|'| ))?

EDIT

I was being wrong with my local test. I was loading up a big string then appending stuff to it which makes .NET Framework dizzy and then give an OOM exception during the RegEx instead of during string operations (or randomly, so ignore the previous stuff I've said).

This is a .NET Framework 2.0 application.

Answer 1

Without seeing your Regex, I don't know for sure but sometimes you can get problems like this because your matches are Greedy instead of Lazy.

The Regex engine has to store lots of information internally and Greedy matches can end up causing the Regex to select large sections of your 800k string, many times over.

There's some good information about this over here.