scrapy.pipelines.files.FilesPipeline

class documentation

class FilesPipeline(MediaPipeline):

Known subclasses: scrapy.pipelines.images.ImagesPipeline

Abstract pipeline that implement the file downloading

This pipeline tries to minimize network transfers and file processing, doing stat of the files and determining if file is new, uptodate or expired.

new files are those that pipeline never processed and needs to be: downloaded from supplier site the first time.
uptodate files are the ones that the pipeline processed and are still: valid files.
expired files are those that pipeline already processed but the last: modification was made long time ago, so a reprocessing is recommended to refresh it in case of change.

Class Method	`from_settings`	Undocumented
Method	`file_path`	Returns the path where downloaded media should be stored
Method	`get_media_requests`	Returns the media requests to download
Method	`item_completed`	Called per item when all media requests has been processed
Constant	`DEFAULT_FILES_RESULT_FIELD`	Undocumented
Constant	`DEFAULT_FILES_URLS_FIELD`	Undocumented
Constant	`EXPIRES`	Undocumented
Constant	`MEDIA_NAME`	Undocumented
Constant	`STORE_SCHEMES`	Undocumented
Method	`__init__`	Undocumented
Method	`_get_store`	Undocumented
Method	`file_downloaded`	Undocumented
Method	`inc_stats`	Undocumented
Method	`media_downloaded`	Handler for success downloads
Method	`media_failed`	Handler for failed downloads
Method	`media_to_download`	Check request before starting download
Instance Variable	`expires`	Undocumented
Instance Variable	`FILES_RESULT_FIELD`	Undocumented
Instance Variable	`files_result_field`	Undocumented
Instance Variable	`FILES_URLS_FIELD`	Undocumented
Instance Variable	`files_urls_field`	Undocumented
Instance Variable	`store`	Undocumented

Inherited from MediaPipeline:

Class Method	`from_crawler`	Undocumented
Constant	`LOG_FAILED_RESULTS`	Undocumented
Class	`SpiderInfo`	Undocumented
Method	`_cache_result_and_execute_waiters`	Undocumented
Method	`_check_media_to_download`	Undocumented
Method	`_check_signature`	Undocumented
Method	`_compatible`	Wrapper for overridable methods to allow backwards compatibility
Method	`_handle_statuses`	Undocumented
Method	`_key_for_pipe`	No summary
Method	`_make_compatible`	Make overridable methods of MediaPipeline and subclasses backwards compatible
Method	`_modify_media_request`	Undocumented
Method	`_process_request`	Undocumented
Method	`open_spider`	Undocumented
Method	`process_item`	Undocumented
Instance Variable	`_expects_item`	Undocumented
Instance Variable	`allow_redirects`	Undocumented
Instance Variable	`download_func`	Undocumented
Instance Variable	`handle_httpstatus_list`	Undocumented
Instance Variable	`spiderinfo`	Undocumented

@classmethod
def from_settings(cls, settings):

overridden in scrapy.pipelines.images.ImagesPipeline

Undocumented

def file_path(self, request, response=None, info=None, *, item=None):

overrides scrapy.pipelines.media.MediaPipeline.file_path

overridden in scrapy.pipelines.images.ImagesPipeline

Returns the path where downloaded media should be stored

def get_media_requests(self, item, info):

overrides scrapy.pipelines.media.MediaPipeline.get_media_requests

overridden in scrapy.pipelines.images.ImagesPipeline

Returns the media requests to download

def item_completed(self, results, item, info):

overrides scrapy.pipelines.media.MediaPipeline.item_completed

overridden in scrapy.pipelines.images.ImagesPipeline

Called per item when all media requests has been processed

DEFAULT_FILES_RESULT_FIELD: str =

Undocumented

Value

'files'

DEFAULT_FILES_URLS_FIELD: str =

Undocumented

Value

'file_urls'

EXPIRES: int =

overridden in scrapy.pipelines.images.ImagesPipeline

Undocumented

Value

MEDIA_NAME: str =

overridden in scrapy.pipelines.images.ImagesPipeline

Undocumented

Value

'file'

STORE_SCHEMES =

Undocumented

Value

{'': FSFilesStore,
 'file': FSFilesStore,
 's3': S3FilesStore,
 'gs': GCSFilesStore,
 'ftp': FTPFilesStore}

def __init__(self, store_uri, download_func=None, settings=None):

overrides scrapy.pipelines.media.MediaPipeline.__init__

overridden in scrapy.pipelines.images.ImagesPipeline

Undocumented

def _get_store(self, uri):

Undocumented

def file_downloaded(self, response, request, info, *, item=None):

overridden in scrapy.pipelines.images.ImagesPipeline

Undocumented

def inc_stats(self, spider, status):

Undocumented

def media_downloaded(self, response, request, info, *, item=None):

overrides scrapy.pipelines.media.MediaPipeline.media_downloaded

Handler for success downloads

def media_failed(self, failure, request, info):

overrides scrapy.pipelines.media.MediaPipeline.media_failed

Handler for failed downloads

def media_to_download(self, request, info, *, item=None):

overrides scrapy.pipelines.media.MediaPipeline.media_to_download

Check request before starting download

expires =

overridden in scrapy.pipelines.images.ImagesPipeline

Undocumented

FILES_RESULT_FIELD =

Undocumented

files_result_field =

Undocumented

FILES_URLS_FIELD =

Undocumented

files_urls_field =

Undocumented

store =

Undocumented