Multimodal Agents and Computer-Use (Capstone)
The 2026 frontier product is a multimodal agent that reads screenshots, clicks buttons, navigates web UIs, fills forms, and completes workflows end-to-end. SeeClick and CogAgent (2024) proved the GUI-grounding primitive. Ferret-UI added mobile. ChartAgent introduced visual tool-use for charts. VisualWebArena and AgentVista (2026) are the benchmarks the frontier chases — and even Gemini 3 Pro and Claude Opus 4.7 score 30% on AgentVista's hard tasks. This capstone pulls together every thread of Phase 12: perception (high-res VLM), reasoning (LLM with tool use), grounding (coordinate output), long-horizon memory, and evaluation. Design a multimodal agent loop: perceive → reason → act → observe → repeat. Build a GUI grounding output schema (click coordinates, type text, scroll, drag) the VLM can emit as JSON. Compare screenshot-only agents vs accessibility-tree agents vs hybrid agents. Set up a multimodal agent benchmark evaluation on a small VisualWebArena slice. A booking-site workflow: "find me a flight to Tokyo for April 15, aisle seat under $800, book it." A multimodal agent needs to: Take a screenshot of the browser. Parse the screenshot + URL + goal into a plan. Emit a structured action: click (at x,y), type "Tokyo" (at element E), scroll down, select (radio button). Apply the action to the browser. Observe the new state (next screenshot). Repeat until the task is done. Each step is a…
Multimodal Agents and Computer-Use (Capstone): The 2026 frontier product is a multimodal agent that reads screenshots, clicks buttons, navigates web UIs,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.