Skip to content

class Token

A token object has the properties of the relax primitive.

Alternatively you can use a getter get_<fieldname> to access a property of a token.

The numbers that represent a catcode are the same as in TeX itself, so using this information assumes that you know a bit about TeX's internals. The other numbers and names are used consistently but are not frozen. So, when you use them for comparing you can best query a known primitive or character first to see the values.

Reference:

😱 Types incomplete or incorrect? 🙏 Please contribute!

fields

Token.command

Token.command : integer

A number representing the internal command number (catcode in case of a character), for example: 1 (left_brace), 2 (right_brace), 3 (math_shift), 4 (tab_mark), 6 (mac_param), 7 (sup_mark), 8 (sub_mark), 10 (spacer), 11 (letter), 12 (other), 147 (call).

Token.cmdname

Token.cmdname : TokenCommandName

The type of the command, for example letter.

Token.csname

Token.csname : string?

The associated control sequence (if applicable), for example bigskip.

Token.id

Token.id : integer

The unique id of the token, for example 6876.

Token.tok

Token.tok : integer

The full token number as stored in TeX, for example 536883863.

Token.active

Token.active : boolean

A boolean indicating the active state of the token, for example true.

Token.expandable

Token.expandable : boolean

A boolean indicating if the token (macro) is expandable, for example true.

Token.protected

Token.protected : boolean

A boolean indicating if the token (macro) is protected, for example false.

Token.mode

Token.mode : integer

A number either representing a character or another entity, for example 1007.

Token.index

Token.index : integer?

A number running from 0x0000 upto 0xFFFF indicating a TeX register index. In case of letters the unicode code point, for example 398 = Ǝ.