Computer Use: Claude, OpenAI CUA, Gemini
Three production computer-use models in 2026. All three are vision-based. All three treat screenshots, DOM text, and tool outputs as untrusted input. Only direct user instructions count as permission. Per-step safety services are the norm. Describe Claude computer use: screenshot in, keyboard/mouse commands out, no accessibility API. Name the three models' benchmark numbers on OSWorld / WebArena / Online-Mind2Web. Explain the per-step safety pattern Gemini 2.5 Computer Use documents. Summarize the untrusted-input contract all three models enforce. Desktop and web agents have to see the screen and drive input. Three vendors shipped productions in the past 18 months. Each made different trade-offs on latency, scope, and safety. Know all three before you pick. Claude 3.5 Sonnet, then Claude 4 / 4.5. Public beta. Vision-based: screenshot in, keyboard/mouse commands out. No OS accessibility APIs — Claude reads pixels. Implementation requires three pieces: an agent loop, the computer tool (schema baked into the model, not developer-configurable), a virtual display (Xvfb on Linux). Claude is trained to count pixels from reference points to target locations, producing resolution-independent coordinates. GPT-4o variant trained with RL on GUI interaction. Merged into ChatGPT agent mode on July 17 2025. Benchmark (at launch): OSWorld 38.1%, WebArena 58.1%, WebVoyager 87%. Developer API: computer-use-preview-2025-03-11 via Responses API. Browser-only (13 actions). 70% Online-Mind2Web accuracy. Lower latency than Anthropic and OpenAI at…
Computer Use: Claude, OpenAI CUA, Gemini: Three production computer-use models in 2026. All three are vision-based. All three treat screenshots, DOM text, and…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.