Eval-Driven Agent Development

Run eval cases, gate merges, review with a rubric, and gate prompt and report changes. Python stdlib.

Run the lessons and projects#

Command	What it does
`python3 code/main.py`	Runs the lesson demo. Lessons 38 and 39 write JSON reports
`VERIFY_DEMO_MODE=1 python3 code/main.py --strict`	Gate demo with the demo secret. Every warn becomes a block
`python3 code/main.py --floor <value>`	Gate demo with a different coverage floor
`python3 scripts/project_test.py <project> --init learning-artifacts/<project>`	Makes a learner workspace
`python3 scripts/project_test.py <project> --stage <n> --path learning-artifacts/<project>`	Tests one stage of your workspace
`python3 scripts/project_test.py <project> --all --solution --strict`	Checks the reference solution
`python3 <dir>/cli.py <cases> <baseline> <candidate> --out <json> --markdown <md>`	Prompt diff. Add `--allow-configuration-change` for a model change
`python3 <dir>/cli.py <claims> --out <json> --html <html>`	Report audit. Add `--research-payload` for a report payload

art = Artifacts(task_id="T-001",

acceptance_commands=accept,zero exit means done

feedback=[{"command": accept[0], "exit_code": 0}],

scope_report={"forbidden_writes": [], "off_scope_writes": []},

rule_report=[{"slug": "done/tests-pass", "passed": True}],

coverage_report={"current": 0.84, "previous": 0.85},

head_commit="a1b2c3d")

report = verify(art, strict=False, coverage_floor=0.80)

report.passedFalse on any block finding

record_override(task_id, finding_code, reason, user_id, head_commit)

verify_signature(entry)HMAC check of the override

Setting	Default	Note
`max_rounds`	3	Evaluator-optimizer rounds per case
`regression_threshold`	0.05	`ci_gate` blocks above it
`COVERAGE_FLOOR_DEFAULT`	0.80	Below it is a block
`COVERAGE_REGRESSION_DELTA`	0.01	Larger drop than the last merge blocks
`release_gate` `min_pass`	1.0	Pass fraction in [0,1]
`release_gate` `max_regressions`	0	Integer regression budget
`judge_claim` `threshold`	0.8	Overlap score to count as supported
`compare` `samples`	2000	Bootstrap resamples, `seed=7`

Symptom	Cause	Fix
Eval scores are unreadable	No baseline	Store a last-known-good baseline
Judge passes wrong output	LLM judge without grounding	Ground the judge on external tools
Evals pass, production fails	Over-fitting to evals	Rotate cases
False alarms in CI	Flaky, non-deterministic cases	Pin seeds, snapshot state
`refused to start: VERIFY_OVERRIDE_SECRET is unset`	No override secret	Set the env var, or `VERIFY_DEMO_MODE=1` for the demo
`recording does not match cases`	Case list changed after recording	Make new recordings
`model/settings changed; explicitly allow this experiment`	Baseline and candidate configs differ	Pass `--allow-configuration-change`
`order_requires_review` on a true claim	Lexical rule rejects a paraphrase	A human reviews the claim

Factor	Static benchmarks	Custom offline	Online
Examples	SWE-bench Verified, GAIA, BFCL V4	LLM-as-judge, execution, trajectory	Session replays, guardrail alerts
Use for	Cross-model comparison	Your product shape	Production behavior
Measures	Broad capability	Right patches for your product	Per-step cost and latency
Watch for	Contamination, 32.67% leakage	Judges hallucinate too	Tie each failure to a new case

Task	Code
Unique ids, known check kinds	`validate_cases(cases)`
Check kinds	`contains`, `excludes`, `json`
Score one recorded response	`score_response(response, checks)`
Pair revisions by case id	`compare(cases, baseline, candidate)`
Pair states	`stable_pass`, `regressed`, `improved`
Ship or block	`release_gate(report)`
Bind recording to cases	`case_sha256 == case_digest(cases)`
Load a provenance-bound run	`load_run(raw, cases)`

Task	Code
One claim per sentence	`parse_claims(text)`
Lexical support of one claim	`support(claim, source)`
Judge cites against evidence	`judge_claim(claim, evidence)`
Reasons for a zero score	`negation`, `number`, `uncited`
Precision, recall, coverage	`score_report(text, evidence)`
Score weights	`precision` 0.5, `recall` 0.25
States	`no_evidence`, `needs_review`
Paired bootstrap	`compare(baseline, candidate)`

Why no LLM in the gate?: The gate gives the same verdict for the same artifacts every time.
Who overrides a block?: Only a human, with override_reason and overridden_by.
Why a separate reviewer?: Role separation: different prompt, different inputs, no write access.
Why pair cases by id?: An average can hide a serious regression.
Why is recall null?: Recall and coverage need labels. Without labels they are unavailable.