pygments.lexers.special.RawTokenLexer

class documentation

class RawTokenLexer(Lexer):

Recreate a token stream formatted with the RawTokenFormatter.

Additional options accepted:

compress: If set to "gz" or "bz2", decompress the token stream with the given compression algorithm before lexing (default: "").

Method	`__init__`	Undocumented
Method	`get_tokens`	Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined.
Method	`get_tokens_unprocessed`	Return an iterable of (index, tokentype, value) pairs where "index" is the starting position of the token within the input text.
Class Variable	`aliases`	Undocumented
Class Variable	`filenames`	Undocumented
Class Variable	`mimetypes`	Undocumented
Class Variable	`name`	Undocumented
Instance Variable	`compress`	Undocumented

Inherited from Lexer:

Method	`analyse_text`	No summary
Class Variable	`alias_filenames`	Undocumented
Method	`__repr__`	Undocumented
Method	`add_filter`	Add a new stream filter to this lexer.
Class Variable	`priority`	Undocumented
Instance Variable	`encoding`	Undocumented
Instance Variable	`ensurenl`	Undocumented
Instance Variable	`filters`	Undocumented
Instance Variable	`options`	Undocumented
Instance Variable	`stripall`	Undocumented
Instance Variable	`stripnl`	Undocumented
Instance Variable	`tabsize`	Undocumented

def __init__(self, **options):

overrides pygments.lexer.Lexer.__init__

Undocumented

def get_tokens(self, text):

overrides pygments.lexer.Lexer.get_tokens

Return an iterable of (tokentype, value) pairs generated from text. If unfiltered is set to True, the filtering mechanism is bypassed even if filters are defined.

Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters.

def get_tokens_unprocessed(self, text):

overrides pygments.lexer.Lexer.get_tokens_unprocessed

Return an iterable of (index, tokentype, value) pairs where "index" is the starting position of the token within the input text.

In subclasses, implement this method as a generator to maximize effectiveness.

aliases: list =

overrides pygments.lexer.Lexer.aliases

Undocumented

filenames: list =

overrides pygments.lexer.Lexer.filenames

Undocumented

mimetypes: list[str] =

overrides pygments.lexer.Lexer.mimetypes

Undocumented

name: str =

overrides pygments.lexer.Lexer.name

Undocumented

compress =

Undocumented