asp.net - HttpWebRequestError: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

Question

I have created a sample asp.net application and try to scrape data from my server using httpwebrequest. But some times I got this above error. I have done some searches on google but all are saying that you should add the property "<httpWebRequest useUnsafeHeaderParsing="true" />" in web.config.

This property have 'UNSAFE" word, so I am too worried about this. I can't add this is in my site configuration. Is there any other option to read the response of my scrape URL. Please let me know how can it be possible without "<httpWebRequest useUnsafeHeaderParsing="true" />"

Thanks in advance, Laxmilal Menaria

Answer 1

This is certainly a server problem-- the server is not following the HTTP specification and the .NET client is flagging this as a potential problem. "Unsafe" is I think somewhat of a misnomer. There's not really a big security issue here, only non-compliance with the RFC which is bad but unfortunately not rare.

So, as you found in Google, the way to get around the problem is to apply the following configuration change:

<configuration> 
 <system.net> 
  <settings> 
   <httpWebRequest useUnsafeHeaderParsing="true" /> 
  </settings> 
 </system.net> 
</configuration>