Video-Language Models: Temporal Tokens and Grounding
Video is not a stack of photos. A 5-second clip has causal ordering, action verbs, and event timing that an image model cannot represent. Video-LLaMA (Zhang et al., June 2023) shipped the first open video-LLM with audio-visual grounding. VideoChat and Video-LLaVA scaled the pattern. By 2025 Qwen2.5-VL's TMRoPE closed the gap with frontier proprietary models. Each system solved temporal tokens differently — Q-former per clip, concat-pool per frame, TMRoPE per token. This lesson reads the patterns, builds a uniform-vs-dynamic frame sampler, and evaluates on temporal grounding tasks. Explain why temporal positional encoding changes video VLM performance independently of the vision encoder. Compare uniform, dynamic-FPS, and event-driven frame sampling on tokens-per-second vs grounding accuracy. Describe Q-former-per-clip (Video-LLaMA) vs pooled-per-frame (Video-LLaVA) vs M-RoPE-per-token (Qwen2.5-VL) designs. Name the four video benchmarks: VideoMME, TempCompass, EgoSchema, Video-MMMU. A 1-minute video at 30 FPS is 1800 frames. At 196 visual tokens per frame (ViT-B at 224), that is 352k tokens — larger than any 2024-era LLM context. Three reduction strategies exist: Subsample frames (1-8 FPS depending on content). Pool each frame's patch tokens aggressively (3x3 or 4x4 bilinear pool). Compress via a Q-former that takes a 16-frame clip and outputs 64 tokens. Each trade-off is different. Subsampling loses temporal detail. Pooling loses spatial detail. Q-former loses both a little but saves tokens. Temporal position encoding is…
Video-Language Models: Temporal Tokens and Grounding: Video is not a stack of photos. A 5-second clip has causal ordering, action verbs, and event timing that…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.