WebScraper.ObeyRobotsDotTxt Field

IronWebScraper - The C# Web Scraping Library

WebScraperObeyRobotsDotTxt Field

Causes the WebScraper to always obey /robots.txt directives including url and path restrictions and crawl rates.

Namespace: IronWebScraper
Assembly: IronWebScraper (in IronWebScraper.dll) Version: 4.0.4.25470 (4.0.4.3)

Syntax

public bool ObeyRobotsDotTxt

Public ObeyRobotsDotTxt As Boolean