Agent Harness Engineering

The code around the model. Loop contract, hooks, budgets, tools, gates, sandbox, traces and workbench files.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Defaults and numbers
Troubleshooting
Questions and tips

Source

Harness architecture#

From Lesson 19.20, 19.20 code/main.py, Lesson 19.23, Lesson 19.25, Lesson 19.26, Lesson 19.28

One step of the loop#

From 19.20 code/main.py, Lesson 19.20

Core loop: HarnessLoop._step#

def _step(self) -> PullRequest | SessionResult:

budget_hit = self._check_budget()on every step

if budget_hit is not None:

return budget_hitpause, back to IDLE

if self.state == State.REFLECTING:

self._cursor += 1turns += 1 also

if self._cursor >= len(self._plan):

return self._complete("goal_met")

step = self._plan[self._cursor]

if step.requires_tool:

self.hooks.fire("before_tool_call", {"step": step})

self.budget.tool_calls += 1count before yield

self._transition(State.AWAITING_TOOL)

return PullRequest(reason="tool_call", ...)pull point

From 19.20 code/main.py

Drive the loop#

Task	Code
Start a session	`out = loop.run("<goal>")`
Subscribe to a hook	`loop.hooks.on("<topic>", <fn>)`
Block a tool call	`raise HookAbort("<reason>")`
Return a tool result	`loop.resume({"result": <value>})`
Resume after a budget pause	`loop.resume()`
Set the budget	`Budget(max_turns=<n>, max_tool_calls=<n>)`

From 19.20 code/main.py

Tools, gates and traces#

Task	Code
Register a tool	`registry.register(<name>, <schema>, <fn>)`
Replace a tool	`registry.register(..., override=True)`
Dispatch a call	`await disp.dispatch("<name>", <args>)`
Collapse duplicate calls	`disp.dispatch(..., idempotency_key="<key>")`
Gate a call	`chain.evaluate(<call>, <ctx>)`
Run in the sandbox	`sandbox.run([<exe>, "<arg>"])`
Span a tool call	`builder.span("gen_ai.tool.execution")`

From 19.21 code/main.py, 19.23 code/main.py, 19.25 code/main.py, 19.26 code/main.py, 19.28 code/main.py

Gate chain, then sandbox#

chain = GateChain(gates=(

WhitelistGate(allowed=frozenset({"read_file"})),allow-set

RegexGate.from_strings(patterns=(r"\bsudo\b",)),argv pattern

RecencyGate(window=3),stale observation

BudgetGate(max_tokens=200),ledger total

))

outcome = chain.evaluate(call, ctx)stops at first deny

if not outcome.allow:

refused += 1see outcome.deny_reason

ledger.record(obs)BudgetGate reads it

config = SandboxConfig(project_root=root)

result = Sandbox(config=config).run(["rm", "-rf", "."])

result.deniedTrue: rm on denylist

From 19.25 code/main.py, 19.26 code/main.py

The seven workbench surfaces#

Instructions: Startup rules, forbidden actions, definition of done.
State: Current task, touched files, blockers, next action.
Scope: Allowed files, forbidden files, acceptance criteria.
Feedback: Real command output captured into the loop.
Verification: Tests, lint, smoke run, scope check.
Review: A second pass with a different role.
Handoff: What changed, why, what is left.

From Lesson 14.31

Workbench scripts#

Command	What it does
`bin/install.sh`	Copies the pack into the repo.
`bin/install.sh --force`	Overwrites an existing `AGENTS.md`.
`python3 scripts/init_agent.py`	Probes runtime, deps, env, state age.
`python3 scripts/run_with_feedback.py <cmd>`	Logs one record to `feedback_record.jsonl`.
`python3 scripts/verify_agent.py <task_id>`	Gate verdict. Exits 1 on a block.
`python3 scripts/generate_handoff.py <task_id>`	Writes `handoff.json` and `handoff.md`.

From 14.42 outputs/agent-workbench-pack/bin/install.sh, 14.42 outputs/agent-workbench-pack/scripts/init_agent.py, 14.42 outputs/agent-workbench-pack/scripts/run_with_feedback.py, 14.42 outputs/agent-workbench-pack/scripts/verify_agent.py, 14.42 outputs/agent-workbench-pack/scripts/generate_handoff.py, Lesson 14.42

Defaults#

Setting	Default	Note
`max_turns`	8	Budget. Turns before a pause
`max_tool_calls`	16	Budget. Tool calls per session
`max_wall_seconds`	30.0	Budget. Wall clock seconds
`timeout_ms`	30_000	Per tool. Per-call override wins
`max_attempts`	3	Dispatcher. Timeout and transient only
`concurrency`	8	Dispatcher fan-out semaphore
`DEFAULT_MAX_OUTPUT_BYTES`	64 * 1024	Sandbox cap per stream
`observation_budget`	4000	Tokens. End-to-end run, lesson 29

From 19.20 code/main.py, 19.21 code/main.py, 19.23 code/main.py, 19.26 code/main.py, 19.29 code/main.py

Troubleshooting#

Symptom	Cause	Fix
`illegal transition`	Edge not in the legal map	Change state only in `_transition`
`requires a payload`	`resume()` at a tool pull point	Pass a result or an error
`budget_exceeded:turns`	A budget limit was reached	Inspect, then `resume()`
`kind="schema"` never retries	Schema errors are deterministic	Fix the arguments
A write runs twice	Retry raced an in-flight call	Pass `idempotency_key`
`feedback.null_exit` block	No exit code was captured	Rerun. Do not advance

From 19.20 code/main.py, 19.21 code/main.py, Lesson 19.23, 19.26 code/main.py, 14.42 outputs/agent-workbench-pack/scripts/verify_agent.py

Glossary#

ReAct: Reason + Act. Thought, Action, Observation in one stream.
OTel: OpenTelemetry. GenAI conventions fix the gen_ai.* keys.
JSON-RPC: Error codes -32601, -32602, -32603.
pass@k: 1 - (1 - p)^k over k samples.
HITL: Human in the loop. Low-confidence actions go here.

From Lesson 14.01, 14.01 outputs/skill-agent-loop.md, 19.23 code/main.py, Lesson 19.27, Lesson 19.28, 15.10 code/main.py

Questions and answers#

Why is the budget a yield?: The caller decides to extend and resume, or to close.
Why no retry on a non-idempotent timeout?: The write possibly committed. A retry duplicates it.
Why cheap gates first?: A deny stops the chain before the expensive checks.
Why no LLM judge in the gate?: Same artifacts must give the same verdict.

From Lesson 19.20, Lesson 19.23, Lesson 19.25, Lesson 19.26, Lesson 14.38

Tips#

Change state in one function only. The invariants then live in one place.
Derive the key in the planner: f"{step_id}:{tool_name}:{hash(args)}".
Return tool errors as observation strings, not crashes.
Hooks mutate or abort. Events only record.
Keep AGENTS.md under 50 lines. Load topic docs on demand.
Put forbidden_files in every scope contract.

From Lesson 19.20, Lesson 19.23, Lesson 14.01, Lesson 14.33, Lesson 14.36