Benchmarks: WebArena and OSWorld
WebArena tests web-agent capability across four self-hosted apps. OSWorld tests desktop-agent capability across Ubuntu, Windows, macOS. At release (2023–2024) both showed a big gap between best-in-class agents and humans. The gap is narrowing; the failure modes haven't changed. Describe WebArena's four self-hosted apps and why execution-based evaluation matters. Explain why OSWorld uses real OS screenshots instead of accessibility APIs. Name the two primary OSWorld failure modes: GUI grounding and operational knowledge. Summarize what OSWorld-G and OSWorld-Human add on top of the base benchmark. Generalist agents can call tools. Can they drive a browser across 20 clicks to complete a shopping checkout? Can they configure a Linux box using only keyboard and mouse? These are the questions WebArena and OSWorld answer. 812 long-horizon tasks across four self-hosted web apps: a shopping site, a forum, a GitLab-like dev tool, a business CMS. Plus utilities: map, calculator, scratchpad. Evaluation is execution-based via gym APIs — was the order placed, was the issue closed, was the CMS page updated? At release: best GPT-4 agent hit 14.41% success vs human 78.24%. The self-hosted framing matters — the benchmark is not flaky because the target apps are pinned and reproducible. VisualWebArena — visually grounded tasks where success depends on interpreting images (screenshots as first-class observations). TheAgentCompany (Dec 2024) — adds terminal + coding; more like a…
Benchmarks: WebArena and OSWorld: WebArena tests web-agent capability across four self-hosted apps. OSWorld tests desktop-agent capability across Ubuntu,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.