IronWebScraper Namespace

IronWebScraper

previous page next page

IronWebScraper Namespace

 
Classes
  ClassDescription
Public classCommonUserAgents
Static helper class which lists common web-browser user-agent strings.
Public classHtmlNode
The HtmlNode class represents a single DOM element in a HTML or XML document.
Public classHtmlNodeExtensions
Extension methods for finding elements within with IEnumerable<HtmlNode>
Public classHttpIdentity
A class defining the browsing 'identity' to be used to fetch a given Url. Contains Proxy, UserAgent and Http Header information.
Public classLicense
A public static class used to apply license keys to IronWebScraper.
Public classCode exampleMetaData

A flexible dictionary of object values which can be used to attach your own additional data or objects to any Request. Meta can contain objects of any Type including instances of classes, List and Dictionaries. This meta-data can then be accessed while Paring the Response and even passed forwards to the next Request.

Metadata send might include pagination page numbers, referrer Urls, User Ids etc.

E.g:
Examples
Copy
Request["page-number"] = 2; 

            int pageNumber = Response.Request.Meta.Get<int>("page-number");
Public classRequest
Represents a http request to be made by IronWebScraper
Public classResponse
Represents a http response made by IronWebScraper
Public classCode exampleScrapedData

A flexible dictionary of object values used to conveniently store scraped data of any Type in a key-value dictionary which can be saved as JSON using the Yield method. ScrapedData can hold data objects of any Type, including Classes.

E.g:
Examples
Copy
var Data = new ScrapedData(); 
            Data['title'] = "Page Title";
            Data['date'] = DateTime.Now;
Public classWebScraper
A base class which developers can extend to build custom web-scraping applications.
Enumerations
  EnumerationDescription
Public enumerationWebScraperLogLevel
Level of WebScraper logging to the Console. Because this Enum is a Flag type options can be combined using a pipe: e.g. LogLevel.Critical | LogLevel.ScrapedData
Public enumerationWebScraperThrottle
Throttle remote clients by their host name or by their public IP address.
previous page start next page