scrapy.core.engine.ExecutionEngine

class documentation

class ExecutionEngine:

View In Hierarchy

Undocumented

Method __init__ Undocumented
Method ​_close​_all​_spiders Undocumented
Method ​_download Undocumented
Method ​_downloaded Undocumented
Method ​_finish​_stopping​_engine Undocumented
Method ​_handle​_downloader​_output Undocumented
Method ​_needs​_backout Undocumented
Method ​_next​_request Undocumented
Method ​_next​_request​_from​_scheduler Undocumented
Method ​_spider​_idle No summary
Method close Close the execution engine gracefully.
Method close​_spider Close (cancel) spider and clear all its outstanding requests
Method crawl Undocumented
Method download Undocumented
Method has​_capacity Does the engine have capacity to handle more spiders
Method open​_spider Undocumented
Method pause Pause the execution engine
Method schedule Undocumented
Method spider​_is​_idle Undocumented
Method start Start the execution engine
Method stop Stop the execution engine gracefully
Method unpause Resume the execution engine
Instance Variable ​_closewait Undocumented
Instance Variable ​_spider​_closed​_callback Undocumented
Instance Variable crawler Undocumented
Instance Variable downloader Undocumented
Instance Variable logformatter Undocumented
Instance Variable paused Undocumented
Instance Variable running Undocumented
Instance Variable scheduler​_cls Undocumented
Instance Variable scraper Undocumented
Instance Variable settings Undocumented
Instance Variable signals Undocumented
Instance Variable slot Undocumented
Instance Variable spider Undocumented
Instance Variable start​_time Undocumented
Property open​_spiders Undocumented
def __init__(self, crawler, spider_closed_callback):

Undocumented

def _close_all_spiders(self):

Undocumented

def _download(self, request, spider):

Undocumented

def _downloaded(self, response, slot, request, spider):

Undocumented

@defer.inlineCallbacks
def _finish_stopping_engine(self):

Undocumented

def _handle_downloader_output(self, response, request, spider):

Undocumented

def _needs_backout(self, spider):

Undocumented

def _next_request(self, spider):

Undocumented

def _next_request_from_scheduler(self, spider):

Undocumented

def _spider_idle(self, spider):
Called when a spider gets idle. This function is called when there are no remaining pages to download or schedule. It can be called multiple times. If some extension raises a DontCloseSpider exception (in the spider_idle signal handler) the spider is not closed until the next loop and this function is guaranteed to be called (at least) once again for this spider.
def close(self):

Close the execution engine gracefully.

If it has already been started, stop it. In all cases, close all spiders and the downloader.

def close_spider(self, spider, reason='cancelled'):
Close (cancel) spider and clear all its outstanding requests
def crawl(self, request, spider):

Undocumented

def download(self, request, spider):

Undocumented

def has_capacity(self):
Does the engine have capacity to handle more spiders
@defer.inlineCallbacks
def open_spider(self, spider, start_requests=(), close_if_idle=True):

Undocumented

def pause(self):
Pause the execution engine
def schedule(self, request, spider):

Undocumented

def spider_is_idle(self, spider):

Undocumented

@defer.inlineCallbacks
def start(self):
Start the execution engine
def stop(self):
Stop the execution engine gracefully
def unpause(self):
Resume the execution engine
_closewait =

Undocumented

_spider_closed_callback =

Undocumented

crawler =

Undocumented

downloader =

Undocumented

logformatter =

Undocumented

paused: bool =

Undocumented

running: bool =

Undocumented

scheduler_cls =

Undocumented

scraper =

Undocumented

settings =

Undocumented

signals =

Undocumented

slot =

Undocumented

spider =

Undocumented

start_time =

Undocumented

@property
open_spiders =

Undocumented