scrapy.utils.python

module documentation

This module contains essential stuff that should've come with Python itself ;)
Class ​Mutable​Chain Thin wrapper around itertools.chain, allowing to add iterables "in-place"
Class ​Weak​Key​Cache Undocumented
Function ​_getargspec​_py23 _getargspec_py23(function) -> named tuple ArgSpec(args, varargs, keywords, defaults)
Function binary​_is​_text Returns True if the given data argument (a bytes object) does not contain unprintable control characters.
Function equal​_attributes Compare two objects attributes
Function flatten flatten(sequence) -> list
Function garbage​_collect Undocumented
Function get​_func​_args Return the argument name list of a callable
Function get​_spec No summary
Function global​_object​_name Return full name of a global object.
Function iflatten iflatten(sequence) -> iterator
Function is​_listlike No summary
Function memoizemethod​_noargs Decorator to cache the result of a method (without arguments) using a weak reference to its object
Function re​_rsearch No summary
Function retry​_on​_eintr Run a function and retry it while getting EINTR errors
Function to​_bytes Return the binary representation of text. If text is already a bytes object, return it as-is.
Function to​_native​_str Return str representation of text.
Function to​_unicode Return the unicode representation of a bytes object text. If text is already an unicode object, return it as-is.
Function unique efficient function to uniquify a list preserving item order
Function without​_none​_values Return a copy of iterable with all None entries removed.
Constant ​_BINARYCHARS Undocumented
def _getargspec_py23(func):
_getargspec_py23(function) -> named tuple ArgSpec(args, varargs, keywords,
defaults)

Was identical to inspect.getargspec() in python2, but uses inspect.getfullargspec() for python3 behind the scenes to avoid DeprecationWarning.

>>> def f(a, b=2, *ar, **kw):
...     pass
>>> _getargspec_py23(f)
ArgSpec(args=['a', 'b'], varargs='ar', keywords='kw', defaults=(2,))
def binary_is_text(data):
Returns True if the given data argument (a bytes object) does not contain unprintable control characters.
def equal_attributes(obj1, obj2, attributes):
Compare two objects attributes
def flatten(x):

flatten(sequence) -> list

Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables).

Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, (8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10] >>> flatten(["foo", "bar"]) ['foo', 'bar'] >>> flatten(["foo", ["baz", 42], "bar"]) ['foo', 'baz', 42, 'bar']

def garbage_collect():

Undocumented

def get_func_args(func, stripself=False):
Return the argument name list of a callable
def get_spec(func):

Returns (args, kwargs) tuple for a function >>> import re >>> get_spec(re.match) (['pattern', 'string'], {'flags': 0})

>>> class Test:
...     def __call__(self, val):
...         pass
...     def method(self, val, flags=0):
...         pass
>>> get_spec(Test)
(['self', 'val'], {})

>>> get_spec(Test.method)
(['self', 'val'], {'flags': 0})

>>> get_spec(Test().method)
(['self', 'val'], {'flags': 0})
def global_object_name(obj):

Return full name of a global object.

>>> from scrapy import Request
>>> global_object_name(Request)
'scrapy.http.request.Request'
def iflatten(x):

iflatten(sequence) -> iterator

Similar to .flatten(), but returns iterator instead

def is_listlike(x): 
>>> is_listlike("foo")
False
>>> is_listlike(5)
False
>>> is_listlike(b"foo")
False
>>> is_listlike([b"foo"])
True
>>> is_listlike((b"foo",))
True
>>> is_listlike({})
True
>>> is_listlike(set())
True
>>> is_listlike((x for x in range(3)))
True
>>> is_listlike(range(5))
True
def memoizemethod_noargs(method):
Decorator to cache the result of a method (without arguments) using a weak reference to its object
def re_rsearch(pattern, text, chunk_size=1024):
This function does a reverse search in a text using a regular expression given in the attribute 'pattern'. Since the re module does not provide this functionality, we have to find for the expression into chunks of text extracted from the end (for the sake of efficiency). At first, a chunk of 'chunk_size' kilobytes is extracted from the end, and searched for the pattern. If the pattern is not found, another chunk is extracted, and another search is performed. This process continues until a match is found, or until the whole file is read. In case the pattern wasn't found, None is returned, otherwise it returns a tuple containing the start position of the match, and the ending (regarding the entire text).
@deprecated
def retry_on_eintr(function, *args, **kw):
Run a function and retry it while getting EINTR errors
def to_bytes(text, encoding=None, errors='strict'):
Return the binary representation of text. If text is already a bytes object, return it as-is.
@deprecated('to_unicode')
def to_native_str(text, encoding=None, errors='strict'):
Return str representation of text.
def to_unicode(text, encoding=None, errors='strict'):
Return the unicode representation of a bytes object text. If text is already an unicode object, return it as-is.
def unique(list_, key=(lambda x: x)):
efficient function to uniquify a list preserving item order
def without_none_values(iterable):

Return a copy of iterable with all None entries removed.

If iterable is a mapping, return a dictionary where all pairs that have value None have been removed.

_BINARYCHARS =

Undocumented

Value
{to_bytes(chr(i)) for i in range(32)}-set([b'\x00', b'\t', b'\n', b'\r'])