multithreading - error MSB4166: Child node exited prematurely. Shutting down

Question

Sometimes my build fail with this error.

 0>MSBUILD : error MSB4166: Child node "3" exited prematurely. Shutting down.

It seems to be completely random and I've not been able to reproduce it at will. I'm running VS2010 Win7 x64 MSBuild 4.0 but this problem seems to be platform and OS independent. I'm building solutions in parallel (/m switch + BuildInParallel=True) and I don't want to disable this feature because I'm compiling application containing 800+ projects . Any idea how to solve it?

EDIT: When I installed .NET 4.5 developer preview, error logging was improved in the MSBuild 4.5 and now the error string looks like this:

error MSB4166: Child node "3" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt

I can find error log file in the Temp folder. This is content of MSBuild_*.failure.txt file:

System.InvalidOperationException: BuildEventArgs has formatted message while serializing!
   at Microsoft.Build.Framework.LazyFormattedBuildEventArgs.WriteToStream(BinaryWriter writer)
   at Microsoft.Build.Framework.BuildMessageEventArgs.WriteToStream(BinaryWriter writer)
   at Microsoft.Build.Shared.LogMessagePacketBase.WriteToStream(INodePacketTranslator translator)
   at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.PacketPumpProc()

Answer 1

After a lot of time and research trying to fix this problem I found a solution that worked for me. I am using msbuild with /m and /p:BuildInParallel=true and the build was failing in the CI server. It was always failing in the second componen with the error:

error MSB4166: Child node "3" exited prematurely. Shutting down.

Adding /nodeReuse:false fixed the problem.

This is a good reference: https://blogs.msdn.microsoft.com/msbuild/2007/04/16/node-reuse-in-multiproc-msbuild/