C# keep session id over httpwebrequest

Question

I need to preserve the same session id when navigating over a site's pages using C#.Net (like a crawler). I found a couple of methods, a http sniffer was very handy, to compare what my IE browser was sending (HTTP request) and receiving from the web server (HTTP response), as the important information is in the headers (that are not displayed by the browser). Please don't make confusion between session id which is public from server to browser, and server's session variables which are private to server code (like php).

WebHeaderCollection headerCollection = new WebHeaderCollection();
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
  /* save headers */
  for (int i = 0; i < response.Headers.Count; i++)
  {
     headerCollection.Add(response.Headers.AllKeys[i], response.Headers.Get(i));
  }
  /* save cookies */
  cookieContainer = new CookieContainer();
  foreach (Cookie cookie in response.Cookies)
  {
    cookieContainer.Add(cookie);
  }
}

to make the other GET or POST requests:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
...
/* restore PHPSESSID */
for (int i = 0; i < headerCollection.Count; i++)
{
 string key = headerCollection.GetKey(i);
 if (key == "Set-Cookie")
 {
  key = "Cookie";
 }
 else
 {
  continue;
 }
 string value = headerCollection.Get(i);
 request.Headers.Add(key, value);
}
/* restore cookies */
request.CookieContainer = cookieContainer;
/* complete request */
Stream writeStream = request.GetRequestStream()

My request is to contribute with better code, or additional ideas to make a better crawler session preserving.

Answer 1

If you create a single cookie container and assign that to both your first and second request you won't need to do all that mucking about copying cookies from the response.

When cookies are set by a response the cookie container that is attached the request will receive and store those cookies. So to maintain the same session context between a series of request just maintain a single cookie container instance and use that with all the requests.

Your code becomes:-

cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
  // Do stuff with response
}

then:-

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
...

request.CookieContainer = cookieContainer;
Stream writeStream = request.GetRequestStream()