Transfusion: Autoregressive Text + Diffusion Image in One Transformer
Chameleon and Emu3 bet everything on discrete tokens. They work, but the quantization bottleneck is visible — the image quality plateaus below continuous-space diffusion models. Transfusion (Meta, Zhou et al., August 2024) takes the opposite bet: keep images continuous, drop the VQ-VAE entirely, and train one transformer with two losses. Text tokens get next-token-prediction. Image patches get a flow-matching / diffusion loss. Both objectives optimize the same weights. The architecture underlying Stable Diffusion 3 (MMDiT) is a close cousin. This lesson reads the Transfusion thesis, builds a toy two-loss trainer, and traces the attention mask that lets one transformer do both jobs. Wire a transformer that runs two losses (NTP on text tokens, diffusion MSE on image patches) on one backbone. Explain why bidirectional attention across image patches plus causal attention over text tokens is the right mask choice. Compare Transfusion-style (continuous images, diffusion loss) to Chameleon-style (discrete images, NTP) on compute, quality, and code complexity. Name MMDiT's contribution: modality-specific weights at each block, joint attention at the residual stream. The discrete vs continuous image tokens debate is older than LLMs. Continuous representations (raw pixels, VAE latents) preserve detail. Discrete tokens (VQ indices) fit the transformer's native vocabulary but lose detail at the quantization step. Chameleon / Emu3 went discrete: one loss, one architecture, but image fidelity capped by tokenizer…
Transfusion: Autoregressive Text + Diffusion Image in One Transformer: Chameleon and Emu3 bet everything on discrete tokens. They work, but the quantization…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.