Chameleon and Early-Fusion Token-Only Multimodal Models
Every VLM we have seen so far keeps images and text separate. Visual tokens come from a vision encoder, flow into a projector, then meet text inside the LLM. The vision and text vocabularies never overlap. Chameleon (Meta, May 2024) asked: what if they did? Train a VQ-VAE that turns an image into a sequence of discrete tokens from a shared vocabulary. Every multimodal document is now one sequence — text tokens and image tokens interleaved, a single autoregressive loss. Side effect: the model can generate mixed-modality outputs — alternating text and image tokens in a single inference call. This lesson reads the early-fusion thesis and builds a toy version end to end. Explain why a shared vocabulary + single loss changes what the model can do. Describe how a VQ-VAE tokenizes an image into a discrete sequence compatible with a transformer's next-token objective. Name Chameleon's training-stability tricks: QK-Norm, dropout placement, LayerNorm ordering. Compare Chameleon vs BLIP-2's Q-Former approach and describe when each is the right choice. Adapter-based VLMs (LLaVA, BLIP-2, Qwen-VL) treat text and image as two different things. A text token goes through embed(texttoken); an image goes through visualencoder(image) → projector → ... pseudotokens. The model has two input paths that merge partway in. Three consequences: The LLM can only consume images, not emit them. Output is text…
Chameleon and Early-Fusion Token-Only Multimodal Models: Every VLM we have seen so far keeps images and text separate. Visual tokens come from a vision…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.