World Models & Video Diffusion
A video model that predicts the next seconds of a scene is a world simulator. Condition that prediction on actions and you have a learned game engine. Explain the difference between a pure video generation model (Sora 2) and an action-conditioned world model (Genie 3, DreamerV3). Describe a video DiT: spatio-temporal patches, 3D position encoding, joint attention across (T, H, W) tokens. Trace how a world model plugs into robotics: VLM plans → video model simulates → inverse dynamics emits actions. Pick between Sora 2, Genie 3, Runway GWM-1 Worlds, Wan-Video, and HunyuanVideo for a given use case (creative video, interactive sim, autonomous-driving synthesis). Video generation and world modelling converged in 2026. A model that can generate a coherent minute of video has, in some sense, learned how the world moves: object permanence, gravity, causality, style. If you condition that prediction on actions (walk left, open the door), the video model becomes a learnable simulator that can replace a game engine, a driving simulator, or a robotics environment. The stakes are concrete. Genie 3 generates playable environments from a single image. Runway GWM-1 Worlds synthesises infinite explorable scenes. Sora 2 produces minute-long videos with synchronised audio and modelled physics. NVIDIA Cosmos-Drive, Wayve Gaia-2, and Tesla DrivingWorld generate realistic driving video for autonomous-vehicle training data. The world-model paradigm is quietly taking…
World Models & Video Diffusion: A video model that predicts the next seconds of a scene is a world simulator. Condition that prediction on actions and you…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.