mistletoe.block_tokenizer

module documentation

Block-level tokenizer for mistletoe.
Class ​File​Wrapper Undocumented
Class ​Parse​Buffer A wrapper around builtin list, so that setattr(list, 'loose') is legal.
Function make​_tokens Takes a list of pairs (token_type, read_result) and applies token_type(read_result).
Function tokenize Searches for token_types in iterable.
Function tokenize​_block Returns a list of pairs (token_type, read_result).
def make_tokens(parse_buffer):

Takes a list of pairs (token_type, read_result) and applies token_type(read_result).

Footnotes are already parsed before this point, and span-level parsing is started here.

def tokenize(iterable, token_types):
Searches for token_types in iterable.
Parameters
iterable:listuser input lines to be parsed.
token​_types:lista list of block-level token constructors.
Returns
block-level token instances.
def tokenize_block(iterable, token_types):

Returns a list of pairs (token_type, read_result).

Footnotes are parsed here, but span-level parsing has not started yet.