Capstone Lesson 27: Eval Harness with Fixture Tasks
A coding agent is only as good as the suite of tasks you measure it against. This lesson builds an evaluation harness that takes a folder of fixture tasks, runs each through a candidate agent, scores pass or fail through a deterministic verifier, and aggregates the results into pass@1, pass@k, mean latency, and mean cost. The harness is the source of truth that lets you tell a regression from a refactor. Define a fixture task as a triple of goal, setup, and verifier. Score multiple sample runs per task and compute pass@1 and pass@k. Aggregate latency and cost into mean and 95th-percentile metrics. Wire deterministic verifiers (file diff, exit code, regex match) into reusable functions. Emit a structured JSON report a regression-tracking script can ingest. Three failure modes plague agent benchmarks built without an eval harness. The first is unverified pass. The agent says it fixed the bug, the human glances at the diff, the suite is marked green, and three weeks later the regression test surfaces the same bug. The agent had reasoned plausibly without actually fixing anything. The second is undetected regression. A change to the prompt template makes the agent 4% better on the loud task and 14% worse on the quiet one. Without a goldset and a per-task score, the regression rides into main and surfaces…
Capstone Lesson 27: Eval Harness with Fixture Tasks: A coding agent is only as good as the suite of tasks you measure it against. This lesson builds an…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.