scrapy.robotstxt.RobotParser

Class Method	`from_crawler`	Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend.
Method	`allowed`	Return `True` if `user_agent` is allowed to crawl `url`, otherwise return `False`.

@classmethod
@abstractmethod
def from_crawler(cls, crawler, robotstxt_body):

overridden in scrapy.robotstxt.ProtegoRobotParser, scrapy.robotstxt.PythonRobotParser, scrapy.robotstxt.ReppyRobotParser, scrapy.robotstxt.RerpRobotParser

Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend.

Parameters
crawler:`~scrapy.crawler.Crawler` instance	crawler which made the request
robotstxt_body:bytes	content of a robots.txt_ file.

@abstractmethod
def allowed(self, url, user_agent):

overridden in scrapy.robotstxt.ProtegoRobotParser, scrapy.robotstxt.PythonRobotParser, scrapy.robotstxt.ReppyRobotParser, scrapy.robotstxt.RerpRobotParser

Return True if user_agent is allowed to crawl url, otherwise return False.

Parameters
url:str	Absolute URL
user_agent:str	User agent