pydoctor.epydoc.markup.epytext.Token

class documentation

class Token:

View In Hierarchy

Tokens are an intermediate data structure used while constructing the structuring DOM tree for a formatted docstring. There are five types of Token:

  • Paragraphs
  • Literal blocks
  • Doctest blocks
  • Headings
  • Bullets

The text contained in each Token is stored in the contents variable. The string in this variable has been normalized. For paragraphs, this means that it has been converted into a single line of text, with newline/indentation replaced by single spaces. For literal blocks and doctest blocks, this means that the appropriate amount of leading whitespace has been removed from each line.

Each Token has an indentation level associated with it, stored in the indent variable. This indentation level is used by the structuring procedure to assemble hierarchical blocks.

Method __init__ Create a new Token.
Method __repr__ No summary
Method to​_dom
Constant BULLET The tag value for bullet Tokens. This tag value is also used for field tag Tokens, since fields function syntactically the same as list items.
Constant DTBLOCK The tag value for doctest Tokens.
Constant HEADING The tag value for heading Tokens.
Constant LBLOCK The tag value for literal Tokens.
Constant PARA The tag value for paragraph Tokens.
Instance Variable contents The normalized text contained in this Token.
Instance Variable indent The indentation level of this Token (in number of leading spaces). A value of None indicates an unknown indentation; this is used for list items and fields that begin with one-line paragraphs.
Instance Variable level The heading-level of this Token if it is a heading; None, otherwise. Valid heading levels are 0, 1, and 2.
Instance Variable startline The line on which this Token begins. This line number is only used for issuing errors.
Instance Variable tag This Token's type. Possible values are Token.PARA (paragraph), Token.LBLOCK (literal block), Token.DTBLOCK (doctest block), Token.HEADINGC, and Token.BULLETC.
def __init__(self, tag, startline, contents, indent, level=None):
Create a new Token.
Parameters
tag:strThe type of the new Token.
startline:intThe line on which the new Token begins.
contents:strThe normalized contents of the new Token.
indent:Optional[int]The indentation of the new Token (in number of leading spaces). A value of None indicates an unknown indentation.
level:Optional[int]The heading-level of this Token if it is a heading; None, otherwise.
def __repr__(self):
Returns
string

the formal representation of this Token. Tokens have formal representaitons of the form:

    <Token: para at line 12>
def to_dom(self):
Returns
Elementa DOM representation of this Token.
BULLET: string =
The tag value for bullet Tokens. This tag value is also used for field tag Tokens, since fields function syntactically the same as list items.
Value
'bullet'
DTBLOCK: string =
The tag value for doctest Tokens.
Value
'doctestblock'
HEADING: string =
The tag value for heading Tokens.
Value
'heading'
LBLOCK: string =
The tag value for literal Tokens.
Value
'literalblock'
PARA: string =
The tag value for paragraph Tokens.
Value
'para'
contents: string =
The normalized text contained in this Token.
indent: int or None =
The indentation level of this Token (in number of leading spaces). A value of None indicates an unknown indentation; this is used for list items and fields that begin with one-line paragraphs.
level: int or None =
The heading-level of this Token if it is a heading; None, otherwise. Valid heading levels are 0, 1, and 2.
startline: int =
The line on which this Token begins. This line number is only used for issuing errors.
tag: string =
This Token's type. Possible values are Token.PARA (paragraph), Token.LBLOCK (literal block), Token.DTBLOCK (doctest block), Token.HEADINGC, and Token.BULLETC.