scrapy.downloadermiddlewares.robotstxt

This is a middleware to respect robots.txt policies. To activate it you must enable this middleware and enable the ROBOTSTXT_OBEY setting.