Stable Diffusion — Architecture & Fine-Tuning
Stable Diffusion is a DDPM that runs in the latent space of a pretrained VAE, conditioned on text via cross-attention, sampled with a fast deterministic ODE solver, and steered by classifier-free guidance. Trace the five pieces of a Stable Diffusion pipeline: VAE, text encoder, U-Net, scheduler, safety checker — and what each of them actually does. Explain latent diffusion and why training in a 4x64x64 latent space (instead of a 3x512x512 image) reduces compute by 48x without quality loss. Use diffusers to generate images, run image-to-image, inpainting, and ControlNet-guided generation. Fine-tune Stable Diffusion with LoRA on a small custom dataset and load the LoRA adapter at inference. Training a DDPM directly on 512x512 RGB images is expensive. Every training step backprops through a U-Net that sees 3x512x512 = 786,432 input values, and sampling takes 50+ forward passes through that same U-Net. At the quality level of Stable Diffusion 1.5 (released 2022), pixel-space diffusion would need roughly 256 GPU-months of training and 10-30 seconds per image on a consumer GPU. The trick that made open-weight text-to-image practical was latent diffusion (Rombach et al., CVPR 2022). Train a VAE that maps a 3x512x512 image to a 4x64x64 latent tensor and back, then do the diffusion in that latent space. Compute drops by (3512512)/(46464) = 48x. Sampling drops from tens of seconds to…
Stable Diffusion — Architecture & Fine-Tuning: Stable Diffusion is a DDPM that runs in the latent space of a pretrained VAE, conditioned on text via…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.