IronWebScraper - The C# Web Scraping Library
WebScraper Properties |
The WebScraper type exposes the following members.
Properties
Name | Description | |
---|---|---|
FailedUrls |
Gets the number of failed http requests which have exceeded their total maximum number of retries.
| |
HttpRetryAttempts |
The number of times WebScraper will retry a failed URL (normally with a new identity) before considering it non-scrapable.
| |
HttpTimeOut |
Gets or the time after-which a HTTP request will be considered failed or lost.
(non-contactable or Dns unavailable)
| |
MaxHttpConnectionLimit |
Gets or sets the total number of allowed open HTTP requests (threads)
| |
OpenConnectionLimitPerHost |
Gets or sets the allowed number of concurrent HTTP requests (threads) per hostname or IP address. This helps protect hosts against too many requests.
| |
RateLimitPerHost |
Gets or sets minimum polite delay (pause) between request to a given domain or IP address.
| |
SuccessfulFileDownloadCount |
Gets the number of successful http downloads using the DownloadFile and DownloadImage methods..
| |
SuccessfulfulRequestCount |
Gets the number of successful http requests.
| |
ThrottleMode |
Makes the WebSraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses. This is polite in-case multiple scraped domains are hosted on the same machine.
|
See Also