scrapy.Spider

class documentation

class Spider(object_ref):

Known subclasses: scrapy.commands.bench._BenchSpider, scrapy.spiders.crawl.CrawlSpider, scrapy.spiders.feed.CSVFeedSpider, scrapy.spiders.feed.XMLFeedSpider, scrapy.spiders.sitemap.SitemapSpider, scrapy.utils.spider.DefaultSpider

View In Hierarchy

Base class for scrapy spiders. All spiders must inherit from this class.
Class Method from​_crawler Undocumented
Class Method handles​_request Undocumented
Class Method update​_settings Undocumented
Static Method close Undocumented
Method __init__ Undocumented
Method __str__ Undocumented
Method ​_parse Undocumented
Method ​_set​_crawler Undocumented
Method log Log the given message at the given log level
Method make​_requests​_from​_url This method is deprecated.
Method parse Undocumented
Method start​_requests Undocumented
Class Variable custom​_settings Undocumented
Instance Variable crawler Undocumented
Instance Variable name Undocumented
Instance Variable settings Undocumented
Instance Variable start​_urls Undocumented
Property logger Undocumented

Inherited from object_ref:

Method __new__ Undocumented
Class Variable __slots__ Undocumented
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
overridden in scrapy.spiders.crawl.CrawlSpider

Undocumented

@classmethod
def handles_request(cls, request):

Undocumented

@classmethod
def update_settings(cls, settings):

Undocumented

@staticmethod
def close(spider, reason):

Undocumented

def __init__(self, name=None, **kwargs):
overridden in scrapy.spiders.crawl.CrawlSpider, scrapy.spiders.sitemap.SitemapSpider

Undocumented

def __str__(self):

Undocumented

def _parse(self, response, **kwargs):
overridden in scrapy.spiders.crawl.CrawlSpider, scrapy.spiders.feed.CSVFeedSpider, scrapy.spiders.feed.XMLFeedSpider

Undocumented

def _set_crawler(self, crawler):

Undocumented

def log(self, message, level=logging.DEBUG, **kw):

Log the given message at the given log level

This helper wraps a log call to the logger within the spider, but you can use it directly (e.g. Spider.logger.info('msg')) or use any other Python logger too.

def make_requests_from_url(self, url):
This method is deprecated.
def parse(self, response, **kwargs):
overridden in scrapy.commands.bench._BenchSpider

Undocumented

def start_requests(self):
overridden in scrapy.commands.bench._BenchSpider, scrapy.spiders.sitemap.SitemapSpider

Undocumented

custom_settings: Optional[dict] =

Undocumented

crawler =

Undocumented

name =
overridden in scrapy.commands.bench._BenchSpider, scrapy.utils.spider.DefaultSpider

Undocumented

settings =

Undocumented

start_urls: list =

Undocumented

@property
logger =

Undocumented