Video-Language Models and Long Video

Sample frames, budget tokens, encode time, ground events and index long video with timestamped answers.

Scene index pipeline (capstone)#

Factor	Video-LLaMA	Video-LLaVA	Qwen2.5-VL
Frames	16-frame clips at 2 FPS	8 frames	Dynamic FPS
Compressor	Video Q-former, 32 queries	MLP	TMRoPE per token
Time position	1D temporal RoPE	Learned temporal embeddings	Absolute timestamp (t, h, w)
Note	Audio branch, ImageBind	Shared image+video encoder	Best open 2025
Answer to "when"	"early in the clip"	8-16 frame system	"at 4.2 seconds"
Weakness	Fixed clip length	No long video	None stated

Factor	Brute context	Ring attention	Token compression	Agentic retrieval
Example	Gemini 1.5, Gemini 2.5 Pro	LWM, LongVILA	Video-XL, LongVA	VideoAgent
Mechanism	All tokens in one pass	Chunks rotate across devices	One summary token per clip	LLM queries a clip tool
Cost	Millions of tokens	Linear in context	Small context	Only relevant clips
Weakness	Closed, not open-source	Spreads over many devices	Misses exact frames	Retrieval quality
Pick for	>1h general QA	15-60 min	15-60 min	>1h specific query

Budget: tokens(duration_s, fps, per_frame) is duration times FPS times tokens per frame.
Raw: 1 minute at 30 FPS and 196 tokens per frame is 352k tokens.
3x3 pool: 64 tokens per frame. 5 minutes at 1 FPS is 19,200 tokens.
384 native: ~729 tokens per frame. 3x3 pooling gives 81.
30 min: 1 FPS at 81 tokens is 145,800 tokens. 2 FPS is 291,600.
2 hours: 1 FPS at 81 tokens is 583k tokens.
Retrieval: 3 * 30 * 81 + 200 tokens for 3 clips. 99% cheaper.

uni = uniform_sample(duration, 8)step * (i + 0.5)

dyn = dynamic_sample(motion, fps_cap=4, total_budget=12)1 to fps_cap per second

result = evaluate_grounding(predictions, ground)hit when IoU >= tol_iou

result["recall"]hits over ground truth events

model = FramePool(num_classes=10, pretrained=False)ResNet-18 per frame, mean over T

c3d = inflate_2d_to_3d(c2d, time_kernel=3)repeat on time, divide by time_kernel

c21 = Conv2Plus1D(3, 16)spatial, BN, ReLU, temporal

Task	Code
Uniform timestamps	`uniform_sample(duration, n)`
Motion-aware timestamps	`dynamic_sample(motion, fps_cap=4, total_budget=32)`
Score grounding	`evaluate_grounding(predictions, ground_truth, tol_iou=0.3)`
Interval overlap	`iou(a_start, a_end, b_start, b_end)`
Uniform frame indices	`sample_uniform(num_frames_total, T)`
Random contiguous window	`sample_dense(num_frames_total, T, rng=None)`
Lift 2D conv to 3D	`inflate_2d_to_3d(conv2d, time_kernel=3)`
Search three vectors	`multi_vector_search(query, scenes, k=5)`

Command	What it does
`python main.py`	Index demo: RRF search, grounded window
`npm run typecheck`	Type checks the UI half
`npm test`	Job state and store tests
`npm start`	Self-terminating demo
`npm run serve`	HTTP server on :8123
`GET /jobs`	Job summaries as JSON
`GET /job/:id`	Stage timeline, 404 when unknown
Stages	chunk, embed, index, qa

Symptom	Cause	Fix
"open door" equals "close door"	Temporal pooling is order-invariant	Use 3D conv or a transformer
Near-chance on motion classes	Appearance alone is insufficient	Train R(2+1)D, not FramePool
Inflated model breaks batch norm	Activation scale grows	Divide by `time_kernel`
Joint attention runtime balloons	Quadratic in `THW`	Use divided attention
Uniform sampling misses the action	Loses motion peaks	Use dynamic FPS with a max-frame cap
Answer says "early in the clip"	No absolute time position	Use TMRoPE and JSON output
Context overflow on 2h video	583k tokens at 1 FPS	Pool more or use VideoAgent
Miscounts "how many X"	Counting hallucination	Report counting accuracy separately

What does TMRoPE add?: Absolute time per token. It handles uneven dynamic FPS spacing.
Which output format?: Structured JSON with time and event fields.
Which pooling?: 3x3 bilinear for most tasks. 6x6 for agent workflows.
Which path for a 15-minute clip?: Open 72B with native context.
Which path for 2+ hours?: VideoAgent or hierarchical summaries.
78% clip, 82% video?: The model relies on test-time averaging.
Why three vectors per scene?: One pooled vector gives a retrieval regression.