Cheat sheets

Agent Benchmarks: SWE-bench to OSWorld

Pick the benchmark, gate on the right tests, check contamination and compare runs on equal denominators.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Single-agent benchmarks side by side#

FactorSWE-benchGAIAWebArenaOSWorld
TestsCode patchingGeneralist tool useWeb tasks, long horizonReal computer tasks
Size2,294 issues, 12 Python repos466 questions, 300 private812 tasks, 4 apps369 tasks, 3 OSes
Agent inputCodebase at pre-fix commit + issueQuestion. Web, multi-modalitySelf-hosted apps + utilities1920×1080 screenshots
EvaluatorFAIL_TO_PASS and PASS_TO_PASS testsNot givenExecution-based, gym APIsNot given
Best at releaseSWE-agent 12.5%GPT-4 with plugins 15%GPT-4 agent 14.41%Best model 12.24%
HumanNot given92%78.24%72.36%
Main trapContaminationConfusion with code benchmarksStale pinned app versionsGUI grounding

Multi-agent benchmarks side by side#

FactorMARBLECOMMAMedAgentBoardSWE-bench Pro
MeasuresStar, chain, tree, graph topologiesMultimodal asymmetric information4 medical task categoriesUncontaminated, later cutoffs
ScoringMilestone KPIs, partial progressAgent-agent collaborationMulti-agent vs single-LLM vs rules1865 problems, 41 repos
HeadlineGraph best for researchFrontier models struggle vs randomMulti-agent does NOT dominate~23% on Pro vs 70%+ on Verified
LessonCognitive planning adds ~3%Measure before you claimMany systems are over-engineeredVerified alone is not evidence

Use X when#

  • SWE-bench Verified: a code agent. Always report Verified scores.
  • SWE-bench Pro: a contamination reality check. Pro is the current gating test.
  • GAIA: a generalist agent. Use the private leaderboard split.
  • AgentBench: a multi-environment comparison across 8 environments.
  • WebArena Verified: a web agent. Self-host it for continuous evaluation.
  • OSWorld: a desktop agent. Run it in a VM fleet.
  • MARBLE: you compare coordination topologies apples-to-apples.
  • AgentArch: you must justify each layer of an enterprise stack.
  • Custom evals: your product flows. Capture gold trajectories for your top 20 tasks.

SWE-bench gate: run_task#

1Harnesscopy state_before. Count passing FAIL_TO_PASSand PASS_TO_PASS checks.2Agentapply the patch to a copy of the state.3Harnesscount passing checks again on the new state.4Fix gateall FAIL_TO_PASS tests must pass after thepatch.5No-regression gateptp_broke must be 0.6Resultresolved is true only when both gates pass.7Reportresolution rate is resolved tasks over alltasks.

Harness Bench pipeline#

casesNewModelContextModelanswerResultstoreCases JSONID, Prompt, ExpectedCases()strict decode, max1000processEvaluatePolicy()3 policies, callbudgetLeaderboard()equal denominatorsstoreRecordingauthored_fixturemodelHTTPModel()chat-completionsCorrect()exact answercontractcasesNewModelContextModelanswerResultstoreCases JSONID, Prompt,ExpectedCases()strict decode, max1000processEvaluatePolicy()3 policies, callbudgetLeaderboard()equal denominatorsstoreRecordingauthored_fixturemodelHTTPModel()chat-completionsCorrect()exact answercontract

Toy harness metrics in Python#

resolved = (ftp_post == len(task.fail_to_pass)) and (ptp_broke == 0)
score = steps + modalities + toolsGAIA level: <=2 is 1, <=5 is 2, else 3
efficiency = steps / task.gold_stepstrajectory efficiency, 1.0 is gold
delta = r["accuracy_seen"] - r["accuracy_held"]
contam_flag = "*" if delta > 0.1 else " "held-set accuracy is canonical
vs_random = r["accuracy_held"] - rand_baselinebaseline 0.15
r = run_bench(sys, n_seen=40, n_held=160, seed=17)

Run the lessons and the project#

CommandWhat it does
python3 code/main.pyRuns the toy harness of lessons 14.19, 14.20 and 16.24
python3 scripts/project_test.py harness-bench --init /tmp/harness-bench-workCreates the learner workspace
python3 scripts/project_test.py harness-bench --stage 1 --path /tmp/harness-bench-work --strictGrades one stage
python3 scripts/project_test.py harness-bench --all --solution --strictGrades the reference solution
go run . --budget 2 --out /tmp/orchard-budget.jsonReplays Orchard with 2 calls per policy
go run . --cases <cases.json> --endpoint <url> --model <model>Calls a live chat-completions model
--key-env <VAR>Names the environment variable that holds the key

CLI defaults#

SettingDefaultNote
--casesfixtures/orchard-cases.jsonJSON array of cases
--recordingfixtures/orchard-recording.jsonIgnored when --endpoint is set
--budget4Model calls per policy, at most 10000
--timeout30sDeadline per policy, at most 5 minutes
--temperature0HTTP sampling temperature
--max-tokens128HTTP output token ceiling
Case limit1000Files are limited to one MiB
HTTP request capten secondsRedirects and query endpoints are rejected

Troubleshooting#

SymptomCauseFix
Cases returns ErrConflictTwo cases share an IDGive each case a unique ID
Cases returns an error on a valid answerMisspelled Expected keyFix the key. Unknown fields are rejected
Leaderboard returns ErrConflictDataset digest, model digest or budget differRerun all policies on one config
15 min fails against 15 minutesExact answer contractDeclare variants in a separate contract
State is budget-exhaustedA retry spent the budgetLater cases stay in Total. Raise --budget
recording exhausted for requested promptPrompt missing or array used upAdd the exact policy prompt to responses
SWE-bench 50% drops to 35% on SWE-bench+Solution leakage in the issue textReport Verified and SWE-bench+ scores
Delta above 0.1, flag *Seen tasks score higher than held tasksQuote held-set accuracy only

Numbers to remember#

SWE-bench+
32.67% of successful patches leaked solutions. 31.08% had weak tests.
Cutoffs
over 94% of SWE-bench issues predate most model cutoffs.
OSWorld-Human
top agents use 1.4-2.7x more steps than gold.
OSWorld-G
564 grounding-only samples. It splits grounding from planning.
Agent-team gain
~30-40 point delta on SWE-bench Pro.
Coordination tax
appears past ~4 agents on graph topology.
Orchard replay
baseline 1/3 in 3 calls, error-retry 2/3 in 4, evidence 3/3 in 3.
Not measured
cost, adversarial safety, your domain and the worst 1% of tails.

Claim checklist#

Which benchmark and split?
Verified vs Pro is a large gap. A number on the wrong split is worthless.
Contamination check?
Was the benchmark released after the training cutoff? If not, treat with caution.
Which baseline?
Single-LLM, random and prior multi-agent work. Not an untuned copy of the same system.
Significance?
N trials, p-value and a confidence interval. Single runs mislead.
Cost disclosed?
Tokens per task and wall-clock. Report P50/P75/P95, not one number.
Comparable receipts?
Same dataset SHA-256, model config SHA-256 and call budget.
Retry on a wrong answer?
No. Error retries never inspect the expected answer.