scrapy.robotstxt.PythonRobotParser

class PythonRobotParser(RobotParser):

Class Method	`from_crawler`	Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend.
Method	`__init__`	Undocumented
Method	`allowed`	Return `True` if `user_agent` is allowed to crawl `url`, otherwise return `False`.
Instance Variable	`rp`	Undocumented
Instance Variable	`spider`	Undocumented

@classmethod
def from_crawler(cls, crawler, robotstxt_body):

overrides scrapy.robotstxt.RobotParser.from_crawler

Parse the content of a robots.txt_ file as bytes. This must be a class method. It must return a new instance of the parser backend.

Parameters
crawler:`~scrapy.crawler.Crawler` instance	crawler which made the request
robotstxt_body:bytes	content of a robots.txt_ file.

def __init__(self, robotstxt_body, spider):

Undocumented

def allowed(self, url, user_agent):

overrides scrapy.robotstxt.RobotParser.allowed

Return True if user_agent is allowed to crawl url, otherwise return False.

rp =

Undocumented

spider =

Undocumented