DeonticBench

Use this sample to study an agent answering a legal-reasoning question from supplied facts and statutes. It turns the statutes into a searchable corpus, gives the agent reading and calculation tools, and records a submitted answer.

What you supply

Provide a Case containing the question, facts, answer format, and statutes, plus a RunConfig with the policy and enabled tools. build_world() returns the agent, workspace, and task prompt. Supply a scoring callback when collection needs an outcome reward; answers can otherwise be scored outside the rollout.

What to adapt

Start with the corpus construction, answer format, and scorer when adapting this sample to another document-based reasoning task. DeonticEnv handles tool-call accounting and submission, while the context builder controls summarization. These task choices remain separate from FOCUS’s search and learning rules.

The reference below covers the case format, environment, configuration, and assembly helpers.

Case

Represent one DeonticBench problem, its reference answer, and the legal materials needed to solve it.

from focus.environments.deontic import Case

Definition

@dataclass(frozen=True)
class Case

Fields

id: str
domain: str
facts: str
question: str
answer: str
answer_type: str
choices: tuple[str, ...] = ()
statutes: tuple[SourceFile, ...] = ()

DeonticEnv

Apply statute-search and reading actions, account for the round’s tool budget, and grade a submitted answer.

from focus.environments.deontic import DeonticEnv

Definition

class DeonticEnv(ToolEnv[DeonticState])

DeonticEnv.init

def __init__(self, *, tools, budget: int, scorer=None, roll_call_budget: bool=False, **kwargs) -> None

DeonticEnv.on_compaction_failure

def on_compaction_failure(self, state: DeonticState, reasons: tuple[str, ...]) -> DeonticState

DeonticEnv.initial_state

async def initial_state(self) -> DeonticState

DeonticEnv.reward

def reward(self, state: DeonticState, turn: ToolTurn) -> float

DeonticEnv.on_tool_calls

def on_tool_calls(self, state: DeonticState, turn: ToolTurn) -> DeonticState

DeonticState

Track which legal materials the agent has inspected and the remaining tool-call allowance.

from focus.environments.deontic import DeonticState

Definition

@dataclass(frozen=True)
class DeonticState(SubmitFields, RoundBudget)

Fields

calls: tuple[str, ...] = ()
compaction_failure: tuple[str, ...] = ()

RunConfig

Configure the model, tool budgets, context management, and execution choices for a DeonticBench run.

from focus.environments.deontic import RunConfig

Definition

@dataclass(frozen=True)
class RunConfig

Fields

policy: Policy[ToolTurn]
tools: tuple[str, ...]
max_steps: int
call_budget: int
max_prompt_tokens: int
python_timeout: float
no_workspace: bool
no_compact: bool
scorer: Callable[[str], float] | None = None
read_max_chars: int = 0
fold_reply_tokens: int = 4096
compaction_failure: Literal['raise', 'end'] = 'raise'

World

Bundle a prepared legal-reasoning case with its agent, workspace, and prompt.

from focus.environments.deontic import World

Definition

@dataclass(frozen=True)
class World

Fields

agent: Agent[DeonticState, ToolTurn]
workspace: Workspace
prompt: str

build_corpus

Build the searchable corpus from a case’s files and cited statutes.

from focus.environments.deontic import build_corpus

Definition

def build_corpus(case: Case) -> Corpus

build_world

Assemble the corpus, tools, environment, context manager, agent, workspace, and prompt for one legal-reasoning case.

from focus.environments.deontic import build_world

Definition

def build_world(case: Case, cfg: RunConfig) -> World