I experienced this issue after installing .Net 4.5 and am posting a solution here to help others if they stumble into it. @berkayk's answer above worked (exposing the mex on a different port), but I needed to expose both endpoints through the same port.
Assume you have two endpoints, one using netTcpBinding and one using mexTcpBinding.
When using the default bindings, some of the defaults are calculated using OSEnvironmentHelper.ProcessorCount instead of hard coded values as they were in .Net 4.0.
In my case, when using a named netTcpBinding bindingConfiguration, the value supplied for the MaxConnections property was 20. Setting the MaxConnections property on the NetTcpBinding also sets it's TcpTransportBindingElement's MaxPendingConnections property and the TcpTransport's ConnectionPoolSettings.MaxOutboundConnectionsPerEndpoint property to the same value.
When not using a named netTcpBinding bindingConfiguration, and only using the default, the MaxPendingConnections property was calculated by using the following algorithm:
return (12 * OSEnvironmentHelper.ProcessorCount);
The mexTcpBinding's transport also calculated it's MaxPendingConnections property using the above algorithm, so when neither is using a named bindingConfiguration the default values match and there is no issue.
When using a named netTcpBinding bindingConfiguration, the transport's MaxPendingConnections was 20, and the mexTcpBinding's transport's MaxPendingConnections was, on my machine, 96. The difference in values for the MaxPendingConnections between these two endpoints sharing the same port is incompatible.
I also found that this issue occurred with the ListenBacklog set as well.
(I do not know of all the possible conflicting values that may exist.)
To resolve the issue, you can create a custom binding for mex that matches the named bindingConfiguration for netTcpBinding. Example below:
<endpoint binding="netTcpBinding" bindingConfiguration="TestNetTcpBinding"
contract="YourContract" />
<endpoint address="mex" binding="customBinding" bindingConfiguration="TestMexBinding"
contract="IMetadataExchange" />
<bindings>
<customBinding>
<binding name="TestMexBinding">
<tcpTransport maxPendingConnections="20" listenBacklog="20">
<connectionPoolSettings groupName="default" maxOutboundConnectionsPerEndpoint="20" />
</tcpTransport>
</binding>
</customBinding>
<netTcpBinding>
<binding name="TestNetTcpBinding" listenBacklog="20" maxConnections="20"/>
</netTcpBinding>
</bindings>
Or, you can not specify any values that are calculated (like maxConnections and listenBacklog) and accept the defaults (note that MaxOutboundConnectionsPerEndpoint will still retain the default value of 10 as it is not calculated in the same way that the MaxPendingConnections property is):
<binding name="TestNetTcpBinding" ...someOtherProperties except listenBacklog and maxConnections/>
Note: The issue is described here: http://msdn.microsoft.com/en-us/library/aa702636.aspx, but the only solution given is to expose the mex on a different port. Below are some screenshots in reflector that show the difference between .net 4.0 and 4.5 when calculating the MaxPendingConnections defaults: