jinja2.lexer

module documentation

Implements a Jinja / Python combination lexer. The Lexer class is used to do some preprocessing. It filters out invalid operators like the bitshift operators we don't allow in templates. It separates template code and python code in expressions.

Class	`Token`	No class docstring; 0/3 class variable, 2/3 methods documented
Class	`TokenStream`	A token stream is an iterable that yields `Token`s. The parser however does not iterate over it but calls `next` to go one token ahead. The current active token is stored as `current`.
Function	`count_newlines`	Count the number of newline characters in the string. This is useful for extensions that filter a stream.
Constant	`TOKEN_ADD`	Undocumented
Constant	`TOKEN_ASSIGN`	Undocumented
Constant	`TOKEN_BLOCK_BEGIN`	Undocumented
Constant	`TOKEN_BLOCK_END`	Undocumented
Constant	`TOKEN_COLON`	Undocumented
Constant	`TOKEN_COMMA`	Undocumented
Constant	`TOKEN_COMMENT`	Undocumented
Constant	`TOKEN_COMMENT_BEGIN`	Undocumented
Constant	`TOKEN_COMMENT_END`	Undocumented
Constant	`TOKEN_DATA`	Undocumented
Constant	`TOKEN_DIV`	Undocumented
Constant	`TOKEN_DOT`	Undocumented
Constant	`TOKEN_EOF`	Undocumented
Constant	`TOKEN_EQ`	Undocumented
Constant	`TOKEN_FLOAT`	Undocumented
Constant	`TOKEN_FLOORDIV`	Undocumented
Constant	`TOKEN_GT`	Undocumented
Constant	`TOKEN_GTEQ`	Undocumented
Constant	`TOKEN_INITIAL`	Undocumented
Constant	`TOKEN_INTEGER`	Undocumented
Constant	`TOKEN_LBRACE`	Undocumented
Constant	`TOKEN_LBRACKET`	Undocumented
Constant	`TOKEN_LINECOMMENT`	Undocumented
Constant	`TOKEN_LINECOMMENT_BEGIN`	Undocumented
Constant	`TOKEN_LINECOMMENT_END`	Undocumented
Constant	`TOKEN_LINESTATEMENT_BEGIN`	Undocumented
Constant	`TOKEN_LINESTATEMENT_END`	Undocumented
Constant	`TOKEN_LPAREN`	Undocumented
Constant	`TOKEN_LT`	Undocumented
Constant	`TOKEN_LTEQ`	Undocumented
Constant	`TOKEN_MOD`	Undocumented
Constant	`TOKEN_MUL`	Undocumented
Constant	`TOKEN_NAME`	Undocumented
Constant	`TOKEN_NE`	Undocumented
Constant	`TOKEN_OPERATOR`	Undocumented
Constant	`TOKEN_PIPE`	Undocumented
Constant	`TOKEN_POW`	Undocumented
Constant	`TOKEN_RAW_BEGIN`	Undocumented
Constant	`TOKEN_RAW_END`	Undocumented
Constant	`TOKEN_RBRACE`	Undocumented
Constant	`TOKEN_RBRACKET`	Undocumented
Constant	`TOKEN_RPAREN`	Undocumented
Constant	`TOKEN_SEMICOLON`	Undocumented
Constant	`TOKEN_STRING`	Undocumented
Constant	`TOKEN_SUB`	Undocumented
Constant	`TOKEN_TILDE`	Undocumented
Constant	`TOKEN_VARIABLE_BEGIN`	Undocumented
Constant	`TOKEN_VARIABLE_END`	Undocumented
Constant	`TOKEN_WHITESPACE`	Undocumented
Variable	`float_re`	Undocumented
Variable	`ignore_if_empty`	Undocumented
Variable	`ignored_tokens`	Undocumented
Variable	`integer_re`	Undocumented
Variable	`newline_re`	Undocumented
Variable	`operator_re`	Undocumented
Variable	`operators`	Undocumented
Variable	`reverse_operators`	Undocumented
Variable	`string_re`	Undocumented
Variable	`whitespace_re`	Undocumented
Class	`_Rule`	Undocumented
Class	`Failure`	Class that raises a `TemplateSyntaxError` if called. Used by the `Lexer` to specify known errors.
Class	`Lexer`	Class that implements a lexer for a given environment. Automatically created by the environment class, usually you don't have to do that.
Class	`OptionalLStrip`	A special tuple for marking a point in the state that can have lstrip applied.
Class	`TokenStreamIterator`	The iterator for tokenstreams. Iterate over the stream until the eof token is reached.
Function	`_describe_token_type`	Undocumented
Function	`compile_rules`	Compiles all the rules from the environment into a list of rules.
Function	`describe_token`	Returns a description of the token.
Function	`describe_token_expr`	Like `describe_token` but for token expressions.
Function	`get_lexer`	Return a lexer which is probably cached.
Variable	`_lexer_cache`	Undocumented

def count_newlines(value):

Count the number of newline characters in the string. This is useful for extensions that filter a stream.

Parameters
value:`str`	Undocumented
Returns
`int`	Undocumented

TOKEN_ADD =

Undocumented

Value

intern('add')

TOKEN_ASSIGN =

Undocumented

Value

intern('assign')

TOKEN_BLOCK_BEGIN =

Undocumented

Value

intern('block_begin')

TOKEN_BLOCK_END =

Undocumented

Value

intern('block_end')

TOKEN_COLON =

Undocumented

Value

intern('colon')

TOKEN_COMMA =

Undocumented

Value

intern('comma')

TOKEN_COMMENT =

Undocumented

Value

intern('comment')

TOKEN_COMMENT_BEGIN =

Undocumented

Value

intern('comment_begin')

TOKEN_COMMENT_END =

Undocumented

Value

intern('comment_end')

TOKEN_DATA =

Undocumented

Value

intern('data')

TOKEN_DIV =

Undocumented

Value

intern('div')

TOKEN_DOT =

Undocumented

Value

intern('dot')

TOKEN_EOF =

Undocumented

Value

intern('eof')

TOKEN_EQ =

Undocumented

Value

intern('eq')

TOKEN_FLOAT =

Undocumented

Value

intern('float')

TOKEN_FLOORDIV =

Undocumented

Value

intern('floordiv')

TOKEN_GT =

Undocumented

Value

intern('gt')

TOKEN_GTEQ =

Undocumented

Value

intern('gteq')

TOKEN_INITIAL =

Undocumented

Value

intern('initial')

TOKEN_INTEGER =

Undocumented

Value

intern('integer')

TOKEN_LBRACE =

Undocumented

Value

intern('lbrace')

TOKEN_LBRACKET =

Undocumented

Value

intern('lbracket')

TOKEN_LINECOMMENT =

Undocumented

Value

intern('linecomment')

TOKEN_LINECOMMENT_BEGIN =

Undocumented

Value

intern('linecomment_begin')

TOKEN_LINECOMMENT_END =

Undocumented

Value

intern('linecomment_end')

TOKEN_LINESTATEMENT_BEGIN =

Undocumented

Value

intern('linestatement_begin')

TOKEN_LINESTATEMENT_END =

Undocumented

Value

intern('linestatement_end')

TOKEN_LPAREN =

Undocumented

Value

intern('lparen')

TOKEN_LT =

Undocumented

Value

intern('lt')

TOKEN_LTEQ =

Undocumented

Value

intern('lteq')

TOKEN_MOD =

Undocumented

Value

intern('mod')

TOKEN_MUL =

Undocumented

Value

intern('mul')

TOKEN_NAME =

Undocumented

Value

intern('name')

TOKEN_NE =

Undocumented

Value

intern('ne')

TOKEN_OPERATOR =

Undocumented

Value

intern('operator')

TOKEN_PIPE =

Undocumented

Value

intern('pipe')

TOKEN_POW =

Undocumented

Value

intern('pow')

TOKEN_RAW_BEGIN =

Undocumented

Value

intern('raw_begin')

TOKEN_RAW_END =

Undocumented

Value

intern('raw_end')

TOKEN_RBRACE =

Undocumented

Value

intern('rbrace')

TOKEN_RBRACKET =

Undocumented

Value

intern('rbracket')

TOKEN_RPAREN =

Undocumented

Value

intern('rparen')

TOKEN_SEMICOLON =

Undocumented

Value

intern('semicolon')

TOKEN_STRING =

Undocumented

Value

intern('string')

TOKEN_SUB =

Undocumented

Value

intern('sub')

TOKEN_TILDE =

Undocumented

Value

intern('tilde')

TOKEN_VARIABLE_BEGIN =

Undocumented

Value

intern('variable_begin')

TOKEN_VARIABLE_END =

Undocumented

Value

intern('variable_end')

TOKEN_WHITESPACE =

Undocumented

Value

intern('whitespace')

float_re =

Undocumented

ignore_if_empty =

Undocumented

ignored_tokens =

Undocumented

integer_re =

Undocumented

newline_re =

Undocumented

operator_re =

Undocumented

operators =

Undocumented

reverse_operators =

Undocumented

string_re =

Undocumented

whitespace_re =

Undocumented

def _describe_token_type(token_type):

Undocumented

Parameters
token_type:`str`	Undocumented
Returns
`str`	Undocumented

def compile_rules(environment):

Compiles all the rules from the environment into a list of rules.

Parameters
environment:`Environment`	Undocumented
Returns
`t.List[t.Tuple[str, str]]`	Undocumented

def describe_token(token):

Returns a description of the token.

Parameters
token:`Token`	Undocumented
Returns
`str`	Undocumented

def describe_token_expr(expr):

Like describe_token but for token expressions.

Parameters
expr:`str`	Undocumented
Returns
`str`	Undocumented

def get_lexer(environment):

Return a lexer which is probably cached.

Parameters
environment:`Environment`	Undocumented
Returns
`Lexer`	Undocumented

_lexer_cache: t.MutableMapping[t.Tuple, Lexer] =

Undocumented