Benchmarks: SWE-bench, GAIA, AgentBench
Three benchmarks anchor agent evaluation in 2026. SWE-bench tests code patching. GAIA tests generalist tool use. AgentBench tests multi-environment reasoning. Know their composition, their contamination story, and what they do not measure. Name SWE-bench's test harness (FAILTOPASS) and explain why it gates on unit tests. Explain why SWE-bench Verified (OpenAI, 500 tasks) exists and what it removes. Describe GAIA's design: simple for humans, hard for AI; three difficulty levels. Name AgentBench's eight environments and its primary blocker for open-source LLMs. Summarize the SWE-bench+ contamination finding and its implications. Leaderboards tell you which model wins on one benchmark. They do not tell you: Whether the benchmark is contaminated (solutions in training data, test leakage). Whether the benchmark measures what you care about (code vs browsing vs generalist). Whether the evaluator is robust (AST matching, state checks, human review). Know the three anchoring benchmarks and their failure modes before you quote a number. 2,294 real GitHub issues from 12 popular Python repos. Agent gets: the codebase at the pre-fix commit + natural-language issue description. Agent produces: a patch. Evaluator: apply patch, run the repo's test suite. The patch must flip FAILTOPASS tests (previously failing, now passing) without breaking PASSTOPASS tests. SWE-agent (Yang et al., 2024) hit 12.5% at release by emphasizing agent-computer interfaces (file editor commands, search syntax the model understands). OpenAI,…
Benchmarks: SWE-bench, GAIA, AgentBench: Three benchmarks anchor agent evaluation in 2026. SWE-bench tests code patching. GAIA tests generalist tool use.…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.