Undocumented
| Module | ajaxcrawl |
No module docstring; 0/2 variable, 1/1 function, 1/1 class documented |
| Module | cookies |
No module docstring; 0/1 variable, 1/1 class documented |
| Module | defaultheaders |
DefaultHeaders downloader middleware |
| Module | downloadtimeout |
Download timeout middleware |
| Module | httpauth |
HTTP basic auth downloader middleware |
| Module | httpcache |
Undocumented |
| Module | httpcompression |
No module docstring; 0/1 constant, 1/1 class documented |
| Module | httpproxy |
Undocumented |
| Module | redirect |
No module docstring; 0/1 variable, 1/3 class documented |
| Module | retry |
An extension to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error. |
| Module | robotstxt |
This is a middleware to respect robots.txt policies. To activate it you must enable this middleware and enable the ROBOTSTXT_OBEY setting. |
| Module | stats |
Undocumented |
| Module | useragent |
Set User-Agent header per spider or use a default value from settings |
| Module | decompression |
This module implements the DecompressionMiddleware which tries to recognise and extract the potentially compressed responses that may arrive. |