c# - How/Can I use linq to xml to query huge xml files with reasonable memory consumption?

Question

I've not done much with linq to xml, but all the examples I've seen load the entire XML document into memory.

What if the XML file is, say, 8GB, and you really don't have the option?

My first thought is to use the XElement.Load Method (TextReader) in combination with an instance of the FileStream Class.

QUESTION: will this work, and is this the right way to approach the problem of searching a very large XML file?

Note: high performance isn't required.. i'm trying to get linq to xml to basically do the work of the program i could write that loops through every line of my big file and gathers up, but since linq is "loop centric" I'd expect this to be possible....

Answer 1

Using XElement.Load will load the whole file into the memory. Instead, use XmlReader with the XNode.ReadFrom function, where you can selectively load notes found by XmlReader with XElement for further processing, if you need to. MSDN has a very good example doing just that: http://msdn.microsoft.com/en-us/library/system.xml.linq.xnode.readfrom.aspx

If you just need to search the xml document, XmlReader alone will suffice and will not load the whole document into the memory.