pygments.token

module documentation

Basic token types and the standard tokens.

Unknown Field: copyright
Copyright 2006-2021 by the Pygments team, see AUTHORS.
Unknown Field: license
BSD, see LICENSE for details.

Constant	`STANDARD_TYPES`	Undocumented
Variable	`Token`	Undocumented
Class	`_TokenType`	Undocumented
Function	`is_token_subtype`	Return True if `ttype` is a subtype of `other`.
Function	`string_to_tokentype`	Convert a string into a token type:

STANDARD_TYPES =

Undocumented

Value

{Token: '',
 Text: '',
 Whitespace: 'w',
 Escape: 'esc',
 Error: 'err',
 Other: 'x',
 Keyword: 'k',
...

Token =

Undocumented

def is_token_subtype(ttype, other):

Return True if ttype is a subtype of other.

exists for backwards compatibility. use ttype in other now.

def string_to_tokentype(s):

Convert a string into a token type:

>>> string_to_token('String.Double')
Token.Literal.String.Double
>>> string_to_token('Token.Literal.Number')
Token.Literal.Number
>>> string_to_token('')
Token

Tokens that are already tokens are returned unchanged:

>>> string_to_token(String)
Token.Literal.String