LocBench

Use this sample to study an agent locating code relevant to an issue. The agent investigates a repository with list, grep, and read tools, then submits file or symbol locations. The environment scores those locations against reference labels.

What you supply

Provide a case with the issue, repository, base commit, and reference locations, plus a policy. Prepare a Repository at the case’s base commit and pass it to build_world() with a RunConfig. The task prompt exposes the issue; reference changes are used for evaluation.

What to adapt

Adapt RepositoryTool for a different code browser or LocationSubmit for another location format. The metrics module owns scoring, and the environment owns investigation limits and completion. This sample demonstrates code localization; it does not edit the repository or produce a patch.

The reference below covers repository access, task assembly, submission, and evaluation.

Case

Represent a LocBench issue, its repository revision, and the reference change used to evaluate localization.

from focus.environments.locbench import Case

Definition

@dataclass(frozen=True)
class Case

Fields

id: str
repo: str
base_commit: str
problem_statement: str
gold: Gold

Case.from_row

@classmethod
def from_row(cls, row: Mapping[str, Any]) -> Case

Case.task_prompt

def task_prompt(self) -> str

The sole model-facing dataset projection; no hints, patch, or labels.

Gold

Hold the reference files and symbols used to score a localization answer.

from focus.environments.locbench import Gold

Definition

@dataclass(frozen=True)
class Gold

Fields

files: frozenset[str]
edit_functions: frozenset[str]
added_functions: frozenset[str] = frozenset()

LocBenchEnv

Apply repository-inspection actions and evaluate the agent’s submitted file and symbol locations.

from focus.environments.locbench import LocBenchEnv

Definition

class LocBenchEnv(ToolEnv[LocBenchState])

LocBenchEnv.init

def __init__(self, repo: Repository, gold: Gold, *, call_budget: int=20, maximum: int=MAX_LOCATIONS, roll_call_budget: bool=False, repeat_limit: int=0, fold_horizon: int | None=None) -> None

repeat_limit ends the investigation after that many identical consecutive requests (zero disables it). fold_horizon ends it once task turns plus compactions reach that many decisions.

LocBenchEnv.initial_state

async def initial_state(self) -> LocBenchState

LocBenchEnv.should_finish

def should_finish(self, rollout: RolloutState[LocBenchState]) -> bool

LocBenchEnv.on_compaction_failure

def on_compaction_failure(self, state: LocBenchState, reasons: tuple[str, ...]) -> LocBenchState

LocBenchEnv.step

async def step(self, state: LocBenchState, turn: ToolTurn, *, workspace: Workspace | None=None) -> StepResult[LocBenchState]

LocBenchEnv.on_submit

def on_submit(self, state: LocBenchState, turn: ToolTurn, payload: dict[str, Any]) -> LocBenchState

LocBenchEnv.reward

def reward(self, state: LocBenchState, turn: ToolTurn) -> float

LocBenchEnv.on_tool_calls

def on_tool_calls(self, state: LocBenchState, turn: ToolTurn) -> LocBenchState

LocBenchState

Track the agent’s repository-inspection progress, submitted locations, and task allowance.

from focus.environments.locbench import LocBenchState

Definition

@dataclass(frozen=True)
class LocBenchState(RoundBudget)

Fields

locations: tuple[str, ...] = ()
submitted: bool = False
calls: tuple[str, ...] = ()
last_request: tuple[str, str] | None = None
consecutive_requests: int = 0
compaction_failure: tuple[str, ...] = ()

Repository

Expose repository evidence at a fixed base commit so all branches inspect the same source revision.

from focus.environments.locbench import Repository

Definition

class Repository

Repository.init

def __init__(self, directory: Path, commit: str) -> None

Repository.list

def list(self, path: str | None=None) -> dict[str, Any]

Repository.text

def text(self, path: str) -> str

Repository.read

def read(self, path: str, start: int=1, lines: int=READ_LINES) -> dict[str, Any]

Repository.grep

def grep(self, pattern: str, path: str | None=None, max_hits: int=MAX_HITS) -> dict[str, Any]

RunConfig

Configure model execution, repository tools, budgets, and context management for a localization run.

from focus.environments.locbench import RunConfig

Definition

@dataclass(frozen=True)
class RunConfig

Fields

policy: Policy[ToolTurn]
max_steps: int = 80
call_budget: int = 20
compact: bool = False
max_prompt_tokens: int = 32768
fold_reply_tokens: int = 8192
max_locations: int = MAX_LOCATIONS
repeat_limit: int = 0
compaction_failure: Literal['raise', 'end'] = 'raise'
count_folds_in_horizon: bool = False

World

Bundle a prepared localization case, repository, agent, workspace, and prompt.

from focus.environments.locbench import World

Definition

@dataclass(frozen=True)
class World

Fields

agent: Agent[LocBenchState, ToolTurn]
workspace: NullWorkspace
prompt: str

aggregate

Average localization metrics across cases using each metric’s eligibility count.

from focus.environments.locbench import aggregate

Definition

def aggregate(records: Iterable[dict[str, float | None]]) -> dict[str, Any]

build_world

Assemble the repository tools, environment, context manager, agent, workspace, and prompt for one localization case.

from focus.environments.locbench import build_world

Definition

def build_world(case: Case, repo: Repository, config: RunConfig) -> World

load_cases

Read LocBench cases from a user-supplied JSONL file. Each record provides the problem, repository revision, and private scoring labels; duplicate case IDs are rejected.

from focus.environments.locbench.dataset import load_cases

Definition

def load_cases(path: Path) -> list[Case]

Entry

Describe a file or symbol exposed by the immutable repository view.

from focus.environments.locbench.repository import Entry

Definition

@dataclass(frozen=True)
class Entry

Fields

path: str
mode: str
oid: str

Entry.readable

@property
def readable(self) -> bool

score

Compute per-case localization metrics from predicted files and symbols; an empty submission receives zero reward.

from focus.environments.locbench import score

Definition

def score(locations: Sequence[str], gold: Gold) -> dict[str, float | None]