Cheat sheets

Video, Depth and 3D Vision

Sample frames, model time, render NeRFs and Gaussian splats, predict depth and track objects across frames.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Video models#

2D + pool
CNN per frame, mean over time.
3D conv
(T, H, W) kernels. Models motion.
I3D
inflate 2D kernels along time.
(2+1)D
2D spatial, then 1D temporal conv.
Divided
attention over time, then space.
Uniform
T frames evenly. 2D+pool default.
Dense
random contiguous window. 3D convs.
Clip / video acc
one clip or mean of clips.

From Lesson 04.12

Pick a 3D, depth or tracking model#

NeRF
an MLP is the scene. Hours to train.
3DGS
explicit Gaussians. Minutes to train.
Depth Anything V3
relative, frozen DINOv2.
ZoeDepth
metric. Sensitive to intrinsics.
Marigold
diffusion depth. 10-50 steps.
ByteTrack
cars and people at 30+ fps.
DeepSORT
heavy occlusion, ReID features.
SAM 3.1 Multiplex
crowds of one class.

From Lesson 04.13, Lesson 04.22, Lesson 04.26, Lesson 04.27

Tracking by detection#

detectionspredicted boxesmatchedunmatched detectionsunmatched tracksclientFrame tvideo inputmodelDetectordetections at tHungarianassignmentIoU, cosine,motionUpdatematched tracksstoreTracks up tot-1Kalman predictconstant velocityBirthnew tracksAge unmatcheddelete after Ndetectionspredicted boxesmatchedunmatched detectionsunmatched tracksclientFrame tvideo inputmodelDetectordetections at tHungarianassignmentIoU, cosine,motionUpdatematched tracksstoreTracks up tot-1KalmanpredictconstantvelocityBirthnew tracksAge unmatcheddelete after N

From Lesson 04.27

Formulas#

PointNet
f(P) = \max_{p \in P} \mathrm{MLP}(p). Point order does not change it.
Positional encoding
\sin(2^l \pi p), \cos(2^l \pi p) up to L = 10 levels.
Volume render
C = \sum_i T_i (1 - e^{-\sigma_i \delta_i}) c_i, T_i = e^{-\sum_{j<i} \sigma_j \delta_j}
3DGS shape
\Sigma = R S S^T R^T. On screen \Sigma' = J W \Sigma W^T J^T.
3DGS pixel
C = \sum_i \alpha_i T_i c_i, T_i = \prod_{j<i} (1 - \alpha_j)
Pinhole lift
X = (u - c_x) d / f_x, Y = (v - c_y) d / f_y, Z = d
AbsRel
mean of |d_{pred} - d_{gt}| / d_{gt}. Lower is better.
MOTA
1 - (FN + FP + \mathrm{IDSW}) / GT

From Lesson 04.13, Lesson 04.22, Lesson 04.26, Lesson 04.27

Frames, inflation and (2+1)D#

idx = sample_uniform(num_frames_total, T)even spread
idx = sample_dense(num_frames_total, T)contiguous window
model = FramePool(num_classes=10)ResNet-18 per frame, mean over T
x = torch.randn(2, 8, 3, 224, 224)(N, T, 3, H, W)
conv2d = nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False)
conv3d = inflate_2d_to_3d(conv2d, time_kernel=3)repeat on time, divide by 3
c = Conv2Plus1D(3, 64)spatial, then temporal
c(torch.randn(1, 3, 8, 56, 56))same 3D input as conv3d

From Lesson 04.12, 04.12 code/main.py

Render one NeRF ray#

y = positional_encoding(x, L=10)(5, 3) to (5, 60)
t_vals = torch.linspace(2.0, 6.0, N)N = 64 samples on the ray
delta = torch.cat([t_vals[1:] - t_vals[:-1], torch.full_like(t_vals[:1], 1e10)])
alpha = 1.0 - torch.exp(-sigma * delta)opacity of each sample
trans = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]
weights = alpha * transtransmittance times opacity
rendered = (weights.unsqueeze(-1) * rgb).sum(dim=-2)pixel colour
depth = (weights * t_vals).sum(dim=-1)depth along the ray

From Lesson 04.13, 04.13 code/main.py

Commands and exports#

CommandWhat it does
pip install nerfstudio gsplatInstalls the 3DGS tools
ns-download-data exampleGets an example dataset
ns-train splatfacto --data path/to/dataTrains 3DGS, 10-30 minutes on an RTX 4090
.plyRaw Gaussian cloud. Portable, largest file
.splatQuantised format
KHR_gaussian_splattingglTF standard across viewers
UsdVolParticleField3DGaussianSplatOpenUSD schema
model.track(source, tracker="bytetrack.yaml")YOLOv8 with ByteTrack

From Lesson 04.22, Lesson 04.27

Depth metrics and lifting#

aligned = align_scale_shift(pred, gt)fit a * pred + b for relative depth
abs_rel_error(aligned, gt)0.05-0.1 for production models
delta_accuracy(pred, target, threshold=1.25)0.9+ for SOTA
intr = (320.0, 320.0, 160.0, 120.0)fx, fy, cx, cy
pc = depth_to_point_cloud(depth, intr)(H, W, 3)
pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Large-hf")
out = pipe(image)
depth_np = np.array(out["depth"])PIL grayscale to numpy

From Lesson 04.26, 04.26 code/main.py

IoU and Hungarian association#

iou = bbox_iou(track_boxes, det_boxes)(tracks, detections)
cost = 1 - iou
cost[iou < self.iou_threshold] = 1e6block weak matches
row, col = linear_sum_assignment(cost)minimum total cost
for r, c in zip(row, col):
if cost[r, c] < 1.0:
self.tracks[r].update(det_boxes[c], frame)
self.tracks = [t for t in self.tracks if frame - t.last_frame <= self.max_age]
tracker = SimpleTracker()iou_threshold=0.3, max_age=5
tracks = tracker.step(dets, f)list of (id, bbox)

From Lesson 04.27, 04.27 code/main.py

Defaults and numbers#

SettingDefaultNote
Frames per clip T8, 16, 32, or 64More T, more signal and compute
time_kernel3Inflated weights divided by it
NeRF L10Direction uses L_dir=4
Samples per ray64t from 2.0 to 6.0
SH degree316 coefficients per channel, 48 floats
3DGS iterations~30,000Grows ~100k Gaussians to 1-5M
Tracker iou_threshold0.3With max_age=5
Delta threshold1.25Fraction of pixels within 25%

From Lesson 04.12, Lesson 04.13, Lesson 04.22, Lesson 04.26, Lesson 04.27

Troubleshooting#

SymptomCauseFix
"Open door" equals "close door"Temporal pooling is order-invariantUse 3D conv or (2+1)D
BN breaks after inflationWeights not scaledDivide by time_kernel
NeRF looks blurryNo positional encodingAdd Fourier features
Right order, wrong metresRelative depth modelMetric model plus intrinsics
Wrong depth on glass, mirrorsReflective surfacesDo not trust those pixels
High AbsRel on MiDaS outputScale and shift not alignedalign_scale_shift first
ID switches near crowdsIoU-only associationByteTrack second pass or ReID
Track lost in a short occlusionNo motion modelAdd a Kalman predict step

From Lesson 04.12, Lesson 04.13, Lesson 04.26, Lesson 04.27

Questions#

Why does 3DGS match NeRF quality?
Both use the same compositing equation.
Why start with 2D + pool?
It reuses ImageNet weights. It is often within 5-10 points.
Which tracking metric?
IDF1 for who is who. HOTA for sports and comparisons.
What can one image not give?
Absolute scale without intrinsics, and occluded geometry.
Why keep low-confidence boxes?
ByteTrack matches them second. This recovers short occlusions.

From Lesson 04.12, Lesson 04.22, Lesson 04.26, Lesson 04.27