WebScraper.ObeyRobotsDotTxtForHost Method

IronWebScraper

previous page next page

WebScraperObeyRobotsDotTxtForHost Method

Causes the WebScraper to always obey /robots.txt directives including path restrictions and crawl rates on a domain by domain basis. May be overridden for advanced control.

Namespace:  IronWebScraper
Assembly:  IronWebScraper (in IronWebScraper.dll) Version: 4.0.4.25470 (4.0.4.3)
Syntax
Copy
public virtual bool ObeyRobotsDotTxtForHost(
	string Host
)
Public Overridable Function ObeyRobotsDotTxtForHost ( 
	Host As String
) As Boolean

Parameters

Host
Type: SystemString

Return Value

Type: Boolean
See Also
previous page start next page