c# - HtmlAgilityPack HtmlWeb.Load returning empty Document

Question

I have been using HtmlAgilityPack for the last 2 months in a Web Crawler Application with no issues loading a webpage.

Now when I try to load a this particular webpage, the document OuterHtml is empty, so this test fails

var url = "http://www.prettygreen.com/";
var htmlWeb = new HtmlWeb();
var htmlDoc = htmlWeb.Load(url);
var outerHtml = htmlDoc.DocumentNode.OuterHtml;
Assert.AreNotEqual("", pageHtml);

I can load another page from the site with no problems, such as setting

url = "http://www.prettygreen.com/news/";

In the past I once had an issue with encodings, I played around with htmlWeb.OverrideEncoding and htmlWeb.AutoDetectEncoding with no luck. I have no idea what could be the issue here with this webpage.

Answer 1

It seems this website requires cookies to be enabled. So creating a cookie container for your web request should solve the issue:

var url = "http://www.prettygreen.com/";
var htmlWeb = new HtmlWeb();
htmlWeb.PreRequest += request =>
    {
        request.CookieContainer = new System.Net.CookieContainer();
        return true;
    };
var htmlDoc = htmlWeb.Load(url);
var outerHtml = htmlDoc.DocumentNode.OuterHtml;
Assert.AreNotEqual("", outerHtml);