WebScraper.ObeyRobotsDotTxt Field

IronWebScraper

previous page next page

WebScraperObeyRobotsDotTxt Field

Causes the WebScraper to always obey /robots.txt directives including url and path restrictions and crawl rates.

Namespace:  IronWebScraper
Assembly:  IronWebScraper (in IronWebScraper.dll) Version: 4.0.4.25470 (4.0.4.3)
Syntax
Copy
public bool ObeyRobotsDotTxt
Public ObeyRobotsDotTxt As Boolean

Field Value

Type: Boolean
See Also
previous page start next page