WebScraper.SetSiteSpecificCrawlRateLimit Method

IronWebScraper - The C# Web Scraping Library

WebScraperSetSiteSpecificCrawlRateLimit Method

Set a throttle limit for a specific domain

Namespace: IronWebScraper
Assembly: IronWebScraper (in IronWebScraper.dll) Version: 4.0.4.25470 (4.0.4.3)

Syntax

public void SetSiteSpecificCrawlRateLimit(
	string hostName,
	TimeSpan crawlRate
)

Public Sub SetSiteSpecificCrawlRateLimit ( 
	hostName As String,
	crawlRate As TimeSpan
)

hostName: Type: SystemString
The http host (domain name).
crawlRate: Type: SystemTimeSpan
The maximum frequency of http requests for the given hostName.