Cheat sheets

Embodied VLAs and Computer-Use Agents

Action formats for robots, GUI action schemas, safety gates and bounded agent loops for browsers and desktops.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Key terms#

VLA
one model takes image and instruction and outputs action commands.
Action tokenization
quantize each joint target into 256 bins. Each bin is a vocab ID.
FAST
DCT plus quantize. A 30-step trajectory becomes ~10 tokens.
Flow-matching head
small transformer outputs a 50-step action sequence via rectified flow.
System 1 / System 2
large VLM plans slowly. Small action head acts quickly.
GUI grounding
model outputs (x, y) for the target of an instruction on a screenshot.
Hybrid agent
uses screenshot and accessibility tree together.
Read/write boundary
reading is never consequential. A write needs fresh approval.

VLA action formats#

FactorRT-2OpenVLAπ0GR00T N1
PatternPaLM-X + discrete binLlama 7B + discrete binflow-matching headdual-system humanoid
Action output10 tokens per step256 bins per dimcontinuous 50-step sequencesubgoals, then joint commands
Rate3-5 Hz4-5 Hz on an A100effectively 50 Hz~1 Hz plan, 50-100 Hz act
Web to robot mix~1:1~0.5:1similarsim-to-real data
Weightsclosedopenopenopen

Action tokenizers in code#

tokens = discretize(action, bins=256)[-1,1]^D to D integer bins
idx = int((a + 1) / 2 * (bins - 1))then clamp to 0..bins-1
recovered = undiscretize(tokens, bins=256)bin width = 2/255 ~ 0.0078
fast_tokens = fast_compress(trajectory, keep_coeff=4)per-dim DCT, bins=32
coeffs = dct(series)[:keep_coeff]keep low-frequency terms
continuous_flow_count = 1single head output

Multimodal agent loop#

1Agenttake a screenshot of the browser.2VLMparse screenshot, URL and goal into a plan.3VLMemit one structured JSON action per step.4Safetyassess each action before execution. Rejectunsafe actions.5Wrapperapply_action(state, action) returns the newstate.6Agentobserve the next screenshot. Repeat until done.

GUI action schema#

TaskCode
Click a target"click": ["x", "y", "element_desc"]
Type text"type": ["text", "x", "y"]
Scroll"scroll": ["direction", "amount"]
Drag"drag": ["x0", "y0", "x1", "y1"]
Pick an option"select": ["option_index"]
Go to a URL"navigate": ["url"]
Crop and zoom"screenshot_region": ["x0", "y0", "x1", "y1"]
End the task"done": ["success", "explanation"]

Browser agent receipt#

Observationobservecapture> 0.0005blockedclientTaskname, email, allowedOriginprocessrunAgentmaxSteps = 5processchoosefill, click, done, blockedclientDriverobserve, act, captureprocessinspectPNGpixels.py greenFractionstoreReceiptstatus, trace, visualObservationobservecapture> 0.0005blockedclientTaskname, email,allowedOriginprocessrunAgentmaxSteps = 5processchoosefill, click, done,blockedclientDriverobserve, act, captureprocessinspectPNGpixels.pygreenFractionstoreReceiptstatus, trace, visual

Numbers to know#

Control budget
30 Hz control loop means 33ms budget per action.
Open X-Embodiment
970k trajectories across 22 robots for OpenVLA.
Robot adaptation
LoRA fine-tuning on 100-1000 task-specific demos.
OpenAI CUA at launch
OSWorld 38.1%, WebArena 58.1%, WebVoyager 87%.
Gemini 2.5 Computer Use
browser-only, 13 actions, ~70% Online-Mind2Web.
WebArena-Verified
fixes ~11.3% false negatives. Hard subset has 258 tasks.
AgentVista
frontier models 27-40%. Open models 10-20%.

Run the projects#

CommandWhat it does
python3 scripts/project_test.py browser-agent --init learning-artifacts/browser-agentMake a starter workspace. It fails until you implement it
python3 scripts/project_test.py browser-agent --all --path learning-artifacts/browser-agent --strictGrade all stages of your workspace
node cli.ts --task samples/contact.json --output browser-run.jsonRun the offline form task and save the receipt
node cli.ts --task samples/contact.json --liveDrive real Chromium. Set BROWSE_BIN first
rustc --edition 2021 cli.rs -o desktop-cliBuild the desktop action runner
./desktop-cli samples/actions.tsv desktop-framesRun six TSV actions. Writes PPM frames and trace.txt

Defaults and limits#

SettingDefaultNote
maxSteps5Must be an integer from 1 to 100
max_steps8CLI step budget from the task file
greenFraction0.0005Above it the status is complete
submitLabelSave requestButton must be unique, enabled, not dangerous
max_calls32CLI cap. Controller::new refuses 0 or above 100
Text length200Desktop type refuses empty or control characters
scale1.0Fixture frame. Valid range above 0 to 4.0
Command timeout20000Milliseconds for each gstack browse call

Errors and fixes#

SymptomCauseFix
origin changedPage URL left allowedOriginKeep the run on the allowed loopback origin
missing or ambiguous fieldLabel matches zero or two fieldsUse exact unique labels
completion values disagreedone is true but a value differsDo not trust the success banner
unsafe or ambiguous submitButton is dangerous, disabled or not uniqueGive one safe submit button
stalledObservation repeats after an actionCheck that the driver applies the action
screenshot-errorCapture or PNG check failedCLI exits with status 2. Fix the image
stale frameClick generation is not the last captureCapture again and use its generation
action budget exhaustedCalls reached max_callsRaise the budget or cut actions

Injection defenses#

Pagenaivesanitizerrw_boundary
Benignposts okposts okposts ok
Visible-text injectionposts to /api/exfilcatches itblocks the write
URL-fragment injectionposts to /api/exfilmisses itblocks the write
Weak pointfollows page textdoes not inspect URLsneeds correct content origin

Questions#

Why co-fine-tune?
Robot-only models fail on out-of-distribution instructions.
What counts as permission?
Only direct user instructions. Screenshots and DOM text are untrusted.
Can a filter fix prompt injection?
No. Defenses raise the cost. They do not close the class.

Safety tips#

  • Put joint, velocity and workspace limits outside the VLA as control-layer checks.
  • Run a per-step safety classifier before every action.
  • Ask a human before login, purchase or file delete.
  • Allow only the tools that the task needs.
  • Run browser sessions with scoped credentials only.