Eval-Driven Agent Development
Anthropic's guidance: "start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed." Evaluation is not the last step. It's the outer loop that drives every other choice in Phase 14. Name the three evaluation layers — static benchmarks, custom offline, online production — and what each is for. Explain the evaluator-optimizer tight loop. Describe the 2026 best practice: evals live next to code, run in CI, gate PRs. Connect every Phase 14 lesson to the eval case it generates. Agents pass demos. They fail in production in ways demos cannot predict. Benchmarks answer "is this model broadly capable?" not "is this agent shipping the right patches for my product?" The answer: evaluation at three layers, running continuously, with every guardrail and learned rule mapped to an eval case. Static benchmarks — SWE-bench Verified for code (Lesson 19), WebArena/OSWorld for browsing / desktop (Lesson 20), GAIA for generalist (Lesson 19), BFCL V4 for tool use (Lesson 06). Use for cross-model comparison and regression gating. Contamination is real: SWE-bench+ found 32.67% solution leakage. Always report Verified / +-audited scores. Custom offline evals — your product's shape:. LLM-as-judge (Langfuse, Phoenix, Opik — Lesson 24). Execution-based (run the patch, check tests). Trajectory-based (compare action sequences against gold; OSWorld-Human shows top agents 1.4-2.7x over gold). Online evals —…
Eval-Driven Agent Development: Anthropic's guidance: "start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.