Video and 3D Generation
Video DiT patches and divided attention, world models, Gaussian splats, NeRF and multi-view 3D. Formulas and fixes.
- Concepts and terms
- Tasks and commands
- Annotated code
- Diagrams
- Comparisons
- Defaults and numbers
- Troubleshooting
- Questions and tips
Video diffusion pipeline#
From Lesson 08.10, Lesson 04.28
Render and fit Gaussian splats#
From Lesson 04.22
Formulas and numbers#
- Covariance
- \Sigma = R S S^T R^T from the quaternion and the scales.
- Projection
- \Sigma' = J W \Sigma W^T J^T, a 2 × 2 covariance.
- Composite
- C = \sum_i \alpha_i T_i c_i with T_i = \prod_{j < i} (1 - \alpha_j). NeRF uses the same equation.
- SH colour
- degree L gives (L+1)^2 terms per channel. Degree 3 gives 16 terms.
- Gaussian size
- about 60 floats. Five million Gaussians use 240 MB.
- Video tokens
- (T / P_t) \cdot (H / P_h) \cdot (W / P_w)
- Divided attention
- (HW) \cdot T^2 + T \cdot (HW)^2 instead of O((THW)^2).
- Raw video
- 10 s of 1080p at 24 fps is 240 frames, about 1.5 GB.
From Lesson 04.22, Lesson 04.28, Lesson 08.10
Tiny video DiT#
vid = torch.randn(1, 4, 8, 16, 16)(N, C, T, H, W)model = TinyVideoDiT(in_channels=4, dim=64, depth=2, heads=2)out, grid = model(vid)grid (4, 8, 8), out (1, 256, 32)tokens, grid = self.patch(x)Conv3d, stride equals kernelxt = x.view(n, T, H * W, d).permute(0, 2, 1, 3).reshape(n * H * W, T, d)a, _ = self.time_attn(self.ln1(xt), self.ln1(xt), self.ln1(xt), need_weights=False)xs = xt.view(n, T, H * W, d).reshape(n * T, H * W, d)a, _ = self.space_attn(self.ln2(xs), self.ln2(xs), self.ln2(xs), need_weights=False)xs = xs + self.mlp(self.ln3(xs))MLP after time and spaceFit 2D splats to an image#
model = Splats2D(num_splats=64, image_size=64)opt = torch.optim.Adam(model.parameters(), lr=0.05)for step in range(200): pred = model((64, 64))rasterise_2d inside loss = F.mse_loss(pred, target) opt.zero_grad(); loss.backward(); opt.step()alphas = alphas.clamp(0.0, 0.99)in rasterise_2dorder = torch.argsort(depths)learned depth scalar in 2Dout = out + (T * a)[..., None] * colours_sorted[i][None, None, :]T = T * (1.0 - a)transmittanceFunctions in the lessons#
| Task | Code |
|---|---|
| Count video tokens | count_tokens(T, |
| Add 3D positions | rope_3d(tokens, |
| Time then space block | DividedAttentionBlock(dim= |
| Patch plus time embed | patchify_with_pos(noisy) |
| Measure flicker | frame_deltas(clip) |
| Gaussian density | eval_2d_gaussian(means, |
| Build covariances | R @ S @ R.transpose(-1, |
| View-dependent colour | eval_sh_degree_3(sh_coeffs, |
From 04.28 code/main.py, Lesson 04.28, 04.22 code/main.py, 08.10 code/main.py
nerfstudio and export#
| Command | What it does |
|---|---|
pip install nerfstudio gsplat | Installs the trainer and the CUDA rasteriser |
ns-download-data example | Downloads the example data |
ns-train splatfacto --data path/ | Trains 3DGS. 10-30 minutes on an RTX 4090 |
.ply | Raw Gaussian cloud. Portable, largest file |
.splat | Quantised format |
KHR_gaussian_splatting | glTF standard, portable across viewers |
UsdVolParticleField3DGaussianSplat | OpenUSD 26.03 schema |
From Lesson 04.22
3D representations side by side#
| Factor | NeRF | 3D Gaussian splats | Triplane | Mesh |
|---|---|---|---|---|
| Stores | MLP weights | explicit Gaussians | three 2-D feature grids | surface plus PBR material |
| Render | ray integration | rasterise and alpha-composite | ray-marching plus an MLP | game engine |
| Speed | 100-1000x slower than mesh | ~100 fps at 1080p on a 4090 | LRM gives one in ~5 s | TripoSR gives one in ~1 s |
| Edit | retrain | move a subset of Gaussians | not covered | remesh before shipping |
From Lesson 08.12, Lesson 04.22
World model families#
| Factor | Video generation | Action-conditioned | RL world model |
|---|---|---|---|
| Input | text or image prompt | past frames + action | state + action |
| Output | video frames | next frames | next latent + reward |
| Steer mid-rollout | no | yes, latent actions | yes |
| Example | Sora 2 | Genie 3 | DreamerV3 |
From Lesson 04.28
Defaults in the code#
| Setting | Default | Note |
|---|---|---|
patch_t, patch_h, patch_w | 2 | VideoPatch3D halves T, H and W |
dim | 64 | Token width |
heads | 2 | Each attention in the block |
p_h, p_w in count_tokens | 8 | p_t is 2 |
num_splats | 64 | Splats2D in the code file |
Adam lr | 0.05 | The code file uses 0.08 |
| Alpha clamp | 0.99 | Upper bound per Gaussian |
| Initial Gaussians | ~100k | Seeded from SfM points |
Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
| Frames flicker | Each frame sampled alone | Couple frames through attention or shared noise |
| Out of memory | Full 3D attention | Factorize into spatial and temporal |
| Clips over 10s drift | Errors accumulate | Sliding windows and keyframe anchors |
| Objects vanish or merge | No persistent object model | Run object permanence checks before shipping |
| Blurry 3D fit | Views disagree on structure | Multi-view diffusion with shared attention |
| 10M splats, overfit | No densify and prune control | Use densification and pruning heuristics |
| Mesh has holes | Implicit field topology | Run a remesher before shipping |
ValueError in rope_3d | t_dim+h_dim+w_dim is not D | Split D across the three axes |
From Lesson 08.10, Lesson 08.12, 04.28 code/main.py, Lesson 04.28
Questions#
- Why did 3DGS replace NeRF?
- It renders at 100+ fps, trains in minutes and edits directly.
- Why does 3DGS match NeRF quality?
- Both integrate the same radiance-field equation.
- What limits a video DiT at serve time?
- Memory bandwidth, not FLOPs.
- What improved Sora most?
- About 10x more detailed captions.
- How does the 2026 3D stack work?
- Diffusion makes multi-view images. A 3D fit, often splats, follows.
- How is video quality measured?
- FVD for visuals, CLIPScore per frame for prompt alignment.