scrapy.spidermiddlewares.offsite.OffsiteMiddleware

class documentation

class OffsiteMiddleware:

View In Hierarchy

Undocumented

Class Method from​_crawler Undocumented
Method __init__ Undocumented
Method get​_host​_regex Override this method to implement a different offsite policy
Method process​_spider​_output Undocumented
Method should​_follow Undocumented
Method spider​_opened Undocumented
Instance Variable domains​_seen Undocumented
Instance Variable host​_regex Undocumented
Instance Variable stats Undocumented
@classmethod
def from_crawler(cls, crawler):

Undocumented

def __init__(self, stats):

Undocumented

def get_host_regex(self, spider):
Override this method to implement a different offsite policy
def process_spider_output(self, response, result, spider):

Undocumented

def should_follow(self, request, spider):

Undocumented

def spider_opened(self, spider):

Undocumented

domains_seen: set =

Undocumented

host_regex =

Undocumented

stats =

Undocumented