Flamingo and Gated Cross-Attention for Few-Shot VLMs
DeepMind's Flamingo (2022) did two things before anyone else. It showed a single model could process arbitrarily interleaved sequences of images, videos, and text. And it showed VLMs could learn in-context — give a few-shot prompt with three example (image, caption) pairs and the model captions a new image without any gradient step. The mechanism: gated cross-attention layers, inserted between the frozen LLM's existing layers, with a learned tanh gate that starts at zero so the LLM's text capability is preserved at initialization. This lesson walks Flamingo's Perceiver resampler and gated cross-attention architecture — the ancestor of Gemini's interleaved inputs and Idefics2's visual tokens. Explain how gated cross-attention preserves a frozen LLM's text capability at initialization via tanh(gate) = 0. Walk through a Perceiver resampler: N image patches → K fixed "latent" queries via cross-attention. Describe how Flamingo handles interleaved image-text sequences with causal masking that respects image placement. Reproduce a few-shot multimodal prompt structure (3 image-caption examples then a query image). BLIP-2 feeds 32 visual tokens into a frozen LLM's input layer. Works for one image per prompt. But what if you want to feed many images interleaved with text, as in "here is image A, caption it; here is image B, caption it; now here is image C, caption it"? The LLM's self-attention would need to handle image…
Flamingo and Gated Cross-Attention for Few-Shot VLMs: DeepMind's Flamingo (2022) did two things before anyone else. It showed a single model could process…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.