Causes the WebScraper to always obey /robots.txt directives including path restrictions and crawl rates on a domain by domain basis. May be overridden for advanced control.

public virtual bool ObeyRobotsDotTxtForHost(
	string Host
)

Public Overridable Function ObeyRobotsDotTxtForHost ( 
	Host As String
) As Boolean