LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model
Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world had separate lineages: LLaVA-1.5 for single images, multi-image models like Mantis and VILA, video models like Video-LLaVA and Video-LLaMA. Each won its benchmark and failed at the others. LLaVA-OneVision argued a single curriculum could train one model to dominate all three scenarios, and that the emergent task-transfer effects (single-image skills exported to video, multi-image reasoning exported to single-image) beat the sum of specialists. The recipe is deceptively simple: a visual-token budget that stays constant across scenarios, plus an explicit curriculum that moves from single-image to OneVision (multi-image) to video. This lesson reads the budget, the curriculum, and the emergent behaviors. Design a visual-token budget that holds constant across single-image, multi-image, and video inputs. Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting. Explain why a single model beats specialists at the same parameter count when curriculum is done right. Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent. Image, multi-image, and video each stress a model differently. Single-image wants high-resolution tokens (AnyRes, 2880 visual tokens) to catch OCR and fine detail. Budget per sample: one image, 2880 tokens. Multi-image wants several images at moderate resolution (576 tokens each) so reasoning across images fits in context. Budget per sample: 4-8 images,…
LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model: Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world had separate lineages:…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.