scrapy.utils.defer

module documentation

Helper functions for dealing with Twisted deferreds

Function	`defer_fail`	Same as twisted.internet.defer.fail but delay calling errback until next reactor loop
Function	`defer_result`	Undocumented
Function	`defer_succeed`	Same as twisted.internet.defer.succeed but delay calling callback until next reactor loop
Function	`deferred_f_from_coro_f`	Converts a coroutine function into a function that returns a Deferred.
Function	`deferred_from_coro`	Converts a coroutine into a Deferred, or returns the object as is if it isn't a coroutine
Function	`iter_errback`	Wraps an iterable calling an errback if an error is caught while iterating it.
Function	`maybeDeferred_coro`	Copy of defer.maybeDeferred that also converts coroutines to Deferreds.
Function	`mustbe_deferred`	Same as twisted.internet.defer.maybeDeferred, but delay calling callback/errback to next reactor loop
Function	`parallel`	Execute a callable over the objects in the given iterable, in parallel, using no more than `count` concurrent calls.
Function	`process_chain`	Return a Deferred built by chaining the given callbacks
Function	`process_chain_both`	Return a Deferred built by chaining the given callbacks and errbacks
Function	`process_parallel`	Return a Deferred with the output of all successful calls to the given callbacks

def defer_fail(_failure):

Same as twisted.internet.defer.fail but delay calling errback until next reactor loop

It delays by 100ms so reactor has a chance to go through readers and writers before attending pending delayed calls, so do not set delay to zero.

def defer_result(result):

Undocumented

def defer_succeed(result):

Same as twisted.internet.defer.succeed but delay calling callback until next reactor loop

It delays by 100ms so reactor has a chance to go trough readers and writers before attending pending delayed calls, so do not set delay to zero.

def deferred_f_from_coro_f(coro_f):

Converts a coroutine function into a function that returns a Deferred.

The coroutine function will be called at the time when the wrapper is called. Wrapper args will be passed to it. This is useful for callback chains, as callback functions are called with the previous callback result.

def deferred_from_coro(o):

Converts a coroutine into a Deferred, or returns the object as is if it isn't a coroutine

def iter_errback(iterable, errback, *a, **kw):

Wraps an iterable calling an errback if an error is caught while iterating it.

def maybeDeferred_coro(f, *args, **kw):

Copy of defer.maybeDeferred that also converts coroutines to Deferreds.

def mustbe_deferred(f, *args, **kw):

Same as twisted.internet.defer.maybeDeferred, but delay calling callback/errback to next reactor loop

def parallel(iterable, count, callable, *args, **named):

Execute a callable over the objects in the given iterable, in parallel, using no more than count concurrent calls.

Taken from: https://jcalderone.livejournal.com/24285.html

def process_chain(callbacks, input, *a, **kw):

Return a Deferred built by chaining the given callbacks

def process_chain_both(callbacks, errbacks, input, *a, **kw):

Return a Deferred built by chaining the given callbacks and errbacks

def process_parallel(callbacks, input, *a, **kw):

Return a Deferred with the output of all successful calls to the given callbacks