Tiktoken Chat

Use a tiktoken vocabulary through the chat codec interface for native API token accounting.

TiktokenChatTokenizer

Use a tiktoken encoding through the ChatCodec interface. It renders messages as plain text with role labels for local token accounting.

from focus.backends import ChatCodec, TiktokenChatTokenizer

tokenizer = TiktokenChatTokenizer(encoding="o200k_base")
codec = ChatCodec(tokenizer)

The selected tiktoken vocabulary handles encoding and decoding. Native API backends send tool schemas separately.

Definition

class TiktokenChatTokenizer

Fields

chat_template: str | None = 'tiktoken-plain'

TiktokenChatTokenizer.init

def __init__(self, encoding: str='o200k_base') -> None

TiktokenChatTokenizer.encode

def encode(self, text: str, *, add_special_tokens: bool=True) -> list[int]

Encode text as token IDs. add_special_tokens is accepted but ignored. Special-token spellings in the input are encoded as ordinary text.

TiktokenChatTokenizer.decode

def decode(self, token_ids: Sequence[int], *, skip_special_tokens: bool=False) -> str

Decode token IDs as text. skip_special_tokens is accepted but ignored.

TiktokenChatTokenizer.encode_single_token

def encode_single_token(self, raw: bytes) -> int

Look up the token ID for a token’s bytes. This lets the chat adapter recover exact IDs from per-token response data.

TiktokenChatTokenizer.apply_chat_template

def apply_chat_template(self, conversation: Sequence[Message], *, tools: list[dict[str, Any]] | None=None, tokenize: bool, add_generation_prompt: bool, return_dict: bool, return_tensors: str | None) -> list[int]

Render messages as text with role labels, then encode that text. Tool schemas are sent separately by the native API backend and are omitted from this rendering.

TiktokenChatTokenizer.render_text

def render_text(self, conversation: Sequence[Message], add_generation_prompt: bool=True) -> str

Return the conversation as text with one role-labeled block per message. Use it to inspect the transcript used for local token accounting.