LLaVA and Visual Instruction Tuning
LLaVA (April 2023) is the most copied multimodal architecture on the planet. It replaced BLIP-2's Q-Former with a 2-layer MLP, replaced Flamingo's gated cross-attention with naive token concatenation, and trained on 158k visual-instruction turns generated by GPT-4 from text-only captions. Any practitioner who built a VLM between 2023 and 2026 built some variant of LLaVA. LLaVA-1.5 added AnyRes. LLaVA-NeXT bumped resolution. LLaVA-OneVision unified image, multi-image, and video in one recipe. This lesson reads the recipe, implements the projector, and explains why "simpler won." Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096). Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns. Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns. Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win. BLIP-2's Q-Former (Lesson 12.03) compresses an image to 32 tokens. Clean, efficient, good for benchmarks. But it has two problems. First, the Q-Former is trainable but its loss is not the final task. Stage 1 trains ITC+ITM+ITG. Stage 2 trains LM loss. The queries learn some intermediate representation that the LLM then has to decode. Information is lost in the bottleneck. Second, the Q-Former takes 188M params, and at LLaVA's 2023…
LLaVA and Visual Instruction Tuning: LLaVA (April 2023) is the most copied multimodal architecture on the planet. It replaced BLIP-2's Q-Former with a 2-layer…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.