scrapy.utils.url

module documentation

This module contains general purpose URL functions not found in the standard library.

Some of the functions that used to be imported from this module have been moved to the w3lib.url module. Always import those from there instead.

Function	`_is_filesystem_path`	Undocumented
Function	`_is_posix_path`	Undocumented
Function	`_is_windows_path`	Undocumented
Function	`add_http_if_no_scheme`	Add http as the default scheme if it is missing from the url.
Function	`escape_ajax`	Return the crawleable url according to: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started
Function	`guess_scheme`	Add an URL scheme if missing: file:// for filepath-like input or http:// otherwise.
Function	`parse_url`	Return urlparsed url from the given argument (which could be an already parsed url)
Function	`strip_url`	Strip URL string from some of its components:
Function	`url_has_any_extension`	Undocumented
Function	`url_is_from_any_domain`	Return True if the url belongs to any of the given domains
Function	`url_is_from_spider`	Return True if the url belongs to the given spider

def _is_filesystem_path(string):

Undocumented

def _is_posix_path(string):

Undocumented

def _is_windows_path(string):

Undocumented

def add_http_if_no_scheme(url):

Add http as the default scheme if it is missing from the url.

def escape_ajax(url):

Return the crawleable url according to: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started

>>> escape_ajax("www.example.com/ajax.html#!key=value")
'www.example.com/ajax.html?_escaped_fragment_=key%3Dvalue'
>>> escape_ajax("www.example.com/ajax.html?k1=v1&k2=v2#!key=value")
'www.example.com/ajax.html?k1=v1&k2=v2&_escaped_fragment_=key%3Dvalue'
>>> escape_ajax("www.example.com/ajax.html?#!key=value")
'www.example.com/ajax.html?_escaped_fragment_=key%3Dvalue'
>>> escape_ajax("www.example.com/ajax.html#!")
'www.example.com/ajax.html?_escaped_fragment_='

URLs that are not "AJAX crawlable" (according to Google) returned as-is:

>>> escape_ajax("www.example.com/ajax.html#key=value")
'www.example.com/ajax.html#key=value'
>>> escape_ajax("www.example.com/ajax.html#")
'www.example.com/ajax.html#'
>>> escape_ajax("www.example.com/ajax.html")
'www.example.com/ajax.html'

def guess_scheme(url):

Add an URL scheme if missing: file:// for filepath-like input or http:// otherwise.

def parse_url(url, encoding=None):

Return urlparsed url from the given argument (which could be an already parsed url)

def strip_url(url, strip_credentials=True, strip_default_port=True, origin_only=False, strip_fragment=True):

Strip URL string from some of its components:

strip_credentials removes "user:password@"
strip_default_port removes ":80" (resp. ":443", ":21") from http:// (resp. https://, ftp://) URLs
origin_only replaces path component with "/", also dropping query and fragment components ; it also strips credentials
strip_fragment drops any #fragment component

def url_has_any_extension(url, extensions):

Undocumented

def url_is_from_any_domain(url, domains):

Return True if the url belongs to any of the given domains

def url_is_from_spider(url, spider):

Return True if the url belongs to the given spider