What is HTML Agility Pack?
HTML Agility Pack is an HTML parser, which can extract data out of HTML pages, either by using XPATH language or by using simple search queries (we’ll cover both methods).
HTML and XML
HTML is a markup language same as XML, but still there are differences which make dealing with them a bit different. Basically HTML is a strict structure in terms of node sets and attribute sets, but in general more documents online are not exactly following the proper structure since browsers usually manage to deal with rendering out the document in UI.
However, this makes it more difficult to read them in case you want to from website itself as you cannot apply all XML rules since structure is not strictly followed.
It makes sense to use standard .NET classes which work with XML to parse HTML and in some cases it works, but for dynamic content websites, where content is updated daily or even more often and possibility of errors in structure are great, this might be not so useful.
Faking your request and introduce to website as a browser
You might face problems in the beginning later on if the website discovers that you are actually crawling it. To avoid this you need to modify you request to be the same as the one coming from the browser.
Acquring XPath filter for elements
To obtain element XPath you can use the tool build in GoogleChrome or use add-on called FirePath for Firefox browser. XPath that you get from these tools is not actualy the bet since it does not include filtering by attribute, but you can modify it. Still a lot better than writing the whole XPath from the scratch.
For GoogleChrome there is a nice extension called XPath Helper which generates XPath with including attributes of the elements.
static void Main(string args)
HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
string urlToLoad = "http://techcrunch.com/apps/";
HttpWebRequest request = HttpWebRequest.Create(urlToLoad) as HttpWebRequest;
request.Method = "GET";
/* Sart browser signature */
request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
/* Sart browser signature */
WebResponse response = request.GetResponse();
if (htmlDoc.DocumentNode != null)
var articleNodes = htmlDoc.DocumentNode.SelectNodes("/html/body/div[@role='main']/div/div/div/div/ul/li");
if (articleNodes != null && articleNodes.Any())
foreach (var articleNode in articleNodes)
var titleNode = articleNode.SelectSingleNode("div/div/h2/a");
No #1 Recommended ASP.NET MVC Hosting
ASPHostPortal.com is the leading provider of Windows hosting and affordable ASP.NET MVC Hosting. ASPHostPortal proudly working to help grow the backbone of the Internet, the millions of individuals, families, micro-businesses, small business, and fledgling online businesses. ASPHostPortal has ability to support the latest Microsoft and ASP.NET technology, such as: WebMatrix, WebDeploy, Visual Studio 2015, .NET 5/ASP.NET 4.5.2, ASP.NET MVC 6.0/5.2, Silverlight 6 and Visual Studio Lightswitch, ASPHostPortal guarantees the highest quality product, top security, and unshakeable reliability, carefully chose high-quality servers, networking, and infrastructure equipment to ensure the utmost reliability.
HostForLIFE.eu guarantees 99.9% uptime for their professional ASP.NET MVC hosting and actually implements the guarantee in practice. HostForLIFE.eu is the service are excellent and the features of the web hosting plan are even greater than many hosting. HostForLIFE.eu offer IT professionals more advanced features and the latest technology. Relibility, Stability and Performance of servers remain and TOP priority. Even basic service plans are equipped with standard service level agreements for 99.99% uptime. Advanced options raise the bar to 99.99%. HostForLIFE.eu revolutionized hosting with Plesk Control Panel, a Web-based interface that provides customers with 24×7 access to their server and site configuration tools.
DiscountService.com.au is The Best and Cheap ASP.NET MVC Hosting. DiscountService.com.au was established to cater to an under served market in the hosting industry web hosting for customers who want excellent service. DiscountService.com.au guarantees the highest quality product, top security, and unshakeable reliability, carefully chose high-quality servers, networking, and infrastructure equipment to ensure the utmost reliability. DiscountService.com.au has ability to support the latest Microsoft and ASP.NET technology, such as: WebMatrix, WebDeploy, Visual Studio 2015, .NET 5/ASP.NET 4.5.2, ASP.NET MVC 6.0/5.2, Silverlight 6 and Visual Studio Lightswitch. DiscountService.com.au is devoted to offering the best Windows hosting solution for you.