Codec

Convert text and conversations into the token representation used by a policy.

ChatCodec

Turn text and chat messages into tokens using a tokenizer and its chat template.

from focus.backends import ChatCodec

codec = ChatCodec(tokenizer)
tokens = codec.render([{"role": "user", "content": "Hello"}])

The tokenizer needs a chat template to render a conversation. encode() and decode() convert plain text to and from token IDs.

Definition

class ChatCodec

ChatCodec.init

def __init__(self, tokenizer: HFTokenizer) -> None

ChatCodec.tokenizer

@property
def tokenizer(self) -> HFTokenizer

The tokenizer used to encode text and render conversations.

ChatCodec.supports_chat

@property
def supports_chat(self) -> bool

Whether the tokenizer provides the chat template needed to render conversations.

ChatCodec.encode

def encode(self, text: str, *, add_special_tokens: bool=False) -> Sequence[int]

Convert a text fragment to token IDs. add_special_tokens=False by default, so a fragment can extend an existing prefix; render() adds the chat-template tokens when formatting the conversation.

ChatCodec.decode

def decode(self, tokens: Sequence[int], *, skip_special_tokens: bool=False) -> str

ChatCodec.render

def render(self, messages: Sequence[Message], *, tools: Sequence[dict[str, Any]] | None=None, add_generation_prompt: bool=True) -> tuple[int, ...]

Render the full conversation and OpenAI-style function schemas through the tokenizer’s chat template. Raises ValueError if the tokenizer has no chat template.

HFTokenizer

Specify the tokenizer operations required by ChatCodec, including encoding, decoding, and chat-template application. A compatible tokenizer can implement this protocol without inheriting from it.

from focus.codec import HFTokenizer

Definition

class HFTokenizer(Protocol)

Fields

chat_template: str | None

HFTokenizer.encode

def encode(self, text: str, *, add_special_tokens: bool=True) -> list[int]

HFTokenizer.decode

def decode(self, token_ids: Sequence[int], *, skip_special_tokens: bool=False) -> str

HFTokenizer.apply_chat_template

def apply_chat_template(self, conversation: Sequence[Message], *, tools: list[dict[str, Any]] | None=None, tokenize: bool, add_generation_prompt: bool, return_dict: bool, return_tensors: str | None) -> list[int]

Message

A dictionary representing a chat message, such as a user prompt, assistant reply, or tool observation. Its fields follow the conversation format used by the policy.

from focus.codec import Message

Definition

Message = dict[str, Any]

TextCodec

The minimal tokenizer interface: convert text to token IDs and token IDs back to text. Implement it when adapting a tokenizer for a backend.

from focus.codec import TextCodec

Definition

class TextCodec(Protocol)

TextCodec.encode

def encode(self, text: str) -> Sequence[int]

TextCodec.decode

def decode(self, tokens: Sequence[int]) -> str