scrapy.pqueues

module documentation

Undocumented

Variable logger Undocumented
Class ​Downloader​Aware​Priority​Queue PriorityQueue which takes Downloader activity into account: domains (slots) with the least amount of active downloads are dequeued first.
Class ​Downloader​Interface No class docstring; 0/1 instance variable, 1/4 method documented
Class ​Scrapy​Priority​Queue A priority queue implemented using multiple internal queues (typically, FIFO queues). It uses one internal queue for each priority value. The internal queue must implement the following methods:
Function ​_path​_safe Return a filesystem-safe version of a string text
logger =

Undocumented

def _path_safe(text):

Return a filesystem-safe version of a string text

>>> _path_safe('simple.org').startswith('simple.org')
True
>>> _path_safe('dash-underscore_.org').startswith('dash-underscore_.org')
True
>>> _path_safe('some@symbol?').startswith('some_symbol_')
True