Decides if the WebScraper will accept a given url. My be overridden to apply custom middleware logic.

Picks a random identity from WebScraper.Identities for each request. Add Identities with proxy IP addresses, userAgents, headers, cookies, username and password in your Init Method and add them to the WebScraper.Identities List;

Override this method to create your own logic for non-random selection of a HttpIdentity for each request.

Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Much like DownloadFile except if the file has already been downloaded or exists locally, it will not be re-downloaded.

Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Requests a file to be downloaded from the given Url to the local file-system. Often used for scraping documents, assets and images.

Normally called with an Parse Method of IronWebScraper.WebScraper

Caches web http responses for reuse. This allows WebScraper classes to be modified and restarted without re-downloading previously scraped urls.

Caches web http responses for reuse. This allows WebScraper classes to be modified and restarted without re-downloading previously scrape urls.

A handy shortcut method that fetches the text content from any Url (synchronously).

A handy shortcut method that fetches the text content from any Url (synchronously) as a binary data in a byye array (byte[])

Override this method initialize your web-scraper. Important tasks will be to Request at least one start url... and set allowed/banned domain or url patterns.

Logs the specified message to the console. Logs can be Enabled using the EnableLogging. This function has been exposed and is over-ridable to allow for easy Email and Slack notification integration.

Causes the WebScraper to always obey /robots.txt directives including path restrictions and crawl rates on a domain by domain basis. May be overridden for advanced control.

Override this method to create the default Response handler for your web scraper. If you have multiple page types, you can add additional similar methods.

Internal method to parse binary files downloaded by a webScraper.

Internal method to parse images downloaded by a webScraper.

Request adds a new request to the scrape-job queue using the POST http method.

Request adds a new request to the scrape-job queue using the POST http method.

Request adds a new request to the scrape-job queue using the POST http method.

Request adds a new request to the scrape-job queue using the POST http method.

Request adds a new request to the scrape-job queue using the POST http method.

Request adds a new request to the scrape-job queue using the POST http method.

A key method called from with the Init and Parse Methods. Request adds new requests to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

A key method called from with the Init and Parse Methods. Request adds a new request to the scrape-job queue, and decides which method (e.g. Parse) will be used to parse the Response object.

Retries a Response.

Usually called in a Parse method, this method is useful if a Captcha or error screen was encountered during Html parsing.

Appends any scraped data to a file in the JsonLines format. (1 json object per line). Will save any .Net object of any kind. This method is typically used with IronWebScraper.ScrapedData or developer defined classes for scraped data items. The default filename will follow the pattern "NameSpace.TypeName.jsonl". E.g: IronWebScraper.ScrapedData.jsonl

Appends scraped data to a file in the JsonLines format. (1 json object per line). Automatically ignores duplicates. Will save any .Net object of any kind. This method is typically used with IronWebScraper.ScrapedData or developer defined classes for scraped data items. The default filename will follow the pattern "WorkingDirecory/NameSpace.TypeName.jsonl". E.g: Scrape/IronWebScraper.ScrapedData.jsonl

Set a throttle limit for a specific domain

Starts the WebScraper.

Set CrawlId to make this crawl resumable. Will also resume a previous scrawl with the same CrawlId if it exists.

Giving a CrawlId also causes the WebScraper to auto-save its state every 5 minutes in case of a crash, system failure or power outage. This feature is particularly useful for long running web-scraping tasks, allowing hours, days or even weeks of work to be recovered effortlessly.

Starts the WebScraper Asynchronously. Set CrawlId to make this crawl resumable. Will resume a previous scrawl with the same CrawlId if it exists.

Stops this WebScraper instance graceful. The WebScraper may be restated later with no loss of data by calling Start(CrawlId) or StartAsync(CrawlId)

Retrieves IronWebScraper.ScrapedData objects which were saved using the WebScraper.Scrape method.

Retrieves IronWebScraper.ScrapedData objects which were saved using the WebScraper.Scrape method.

Retrieves native C# objects which were saved using the WebScraper.Scrape method in the JsonLines format.

Retrieves native C# objects which were saved using the WebScraper.Scrape method in the JsonLines format.