Video Generation
An image is a 2-D tensor. A video is a 3-D one. The theory is the same; the compute is 10-100x harder. OpenAI's Sora (Feb 2024) proved it was possible. By 2026 Veo 2, Kling 1.5, Runway Gen-3, Pika 2.0, and WAN 2.2 ship production video from text at 1080p — and the open-weights stack (CogVideoX, HunyuanVideo, Mochi-1, WAN 2.2) is 12 months behind. A 10-second 1080p video at 24fps is 240 frames of 1920×1080×3 pixels. That's 1.5 GB of raw data per clip. Pixel-space diffusion is infeasible. You need: Spatiotemporal compression. A VAE that encodes videos, not frames, into a sequence of spatial-temporal patches. Temporal coherence. Frames need to share content, lighting, and object identity over seconds. The net has to model motion. Compute budget. Video training is 10-100x more expensive than image for the same model size. Conditioning. Text, image (first-frame), audio, or another video. Most production models accept all four. The architecture that solved this is the Diffusion Transformer (DiT) applied to spatiotemporal patches, trained on huge (prompt, caption, video) datasets. Same diffusion loss as Lesson 06. Video diffusion: patchify, DiT, decode Encode the video with a 3D VAE (learned spatiotemporal compression). The latent is shape [Tlatent, Hlatent, Wlatent, Clatent]. Split into patches of size [tp, hp, wp]. For Sora-style models, tp = 1 (per-frame patches) or…
Video Generation: An image is a 2-D tensor. A video is a 3-D one. The theory is the same; the compute is 10-100x harder. OpenAI's Sora (Feb 2024) proved it…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.