package documentation


Module ajaxcrawl No module docstring; 0/2 variable, 1/1 function, 1/1 class documented
Module cookies No module docstring; 0/1 variable, 1/1 class documented
Module defaultheaders DefaultHeaders downloader middleware
Module downloadtimeout Download timeout middleware
Module httpauth HTTP basic auth downloader middleware
Module httpcache Undocumented
Module httpcompression No module docstring; 0/1 constant, 1/1 class documented
Module httpproxy Undocumented
Module redirect No module docstring; 0/1 variable, 1/3 class documented
Module retry An extension to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error.
Module robotstxt This is a middleware to respect robots.txt policies. To activate it you must enable this middleware and enable the ROBOTSTXT_OBEY setting.
Module stats Undocumented
Module useragent Set User-Agent header per spider or use a default value from settings
Module decompression This module implements the DecompressionMiddleware which tries to recognise and extract the potentially compressed responses that may arrive.