Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
660 views
in Technique[技术] by (71.8m points)

c# - HtmlAgilityPack and Authentication

I have a method to get ids and xpaths if given a particular url. How do I pass in the username and password with the request so that I can scrape a url that requires a username and password?

using HtmlAgilityPack;

_web = new HtmlWeb();

internal Dictionary<string, string> GetidsAndXPaths(string url)
{
    var webidsAndXPaths = new Dictionary<string, string>();
    var doc = _web.Load(url);
    var nodes = doc.DocumentNode.SelectNodes("//*[@id]");
    if (nodes == null) return webidsAndXPaths;
    // code to get all the xpaths and ids

Should I use a web request to get the page source and then pass that file into the method above?

var wc = new WebClient();
wc.Credentials = new NetworkCredential("UserName", "Password");
wc.DownloadFile("http://somewebsite.com/page.aspx", @"C:localfile.html");
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

HtmlWeb.Load has a number of overloads, these accept either an instance of NetworkCredential or you can pass in a username and password directly.

Name // Description 
Public method Load(String) //Gets an HTML document from an Internet resource.  
Public method Load(String, String) //Loads an HTML document from an Internet resource.  
Public method Load(String, String, WebProxy, NetworkCredential) //Loads an HTML document from an Internet resource.  
Public method Load(String, String, Int32, String, String) //Loads an HTML document from an Internet resource. 

You do not need to pass in a WebProxy instance, or you can pass in the system default one.

Alternatively you can wire up the HtmlWeb.PreRequest and setup the credentials for the request.

htmlWeb.PreRequest += (request) => {
    request.Credentials = new NetworkCredential(...);
    return true;
};

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...