Open VLM Families: LLaVA, Qwen-VL, InternVL

LLaVA, LLaVA-OneVision, Qwen-VL and InternVL3 side by side: connector, resolution, video, tokens and training.

Families side by side#

Factor	LLaVA-NeXT	LLaVA-OneVision	Qwen-VL family	InternVL3
Vision to LLM	2-layer MLP, GELU	Same MLP projector	Qwen2-VL: 2-layer MLP on merged patches	Native, no align stage
Recipe	Post-hoc on a text LLM	Post-hoc on a text LLM	Post-hoc on a text LLM	Native multimodal pretrain
High resolution	AnyRes: 336 tiles plus thumbnail	AnyRes-9 or AnyRes-4, 384 tiles	Qwen2-VL: any H and W divisible by 28	ViR picks low, medium or high
Video	Per-frame concat	32 frames, 81 tokens each	Dynamic FPS, `<time>` tokens	5% video-text in pretrain
Visual tokens	576 base, 2880 AnyRes	3000-4000 per sample, fixed	Scales with input	256, 576 or 2048+ via ViR
Training	Align, then instruct	Stages SI, OV, TT	Qwen3-VL: stages stress OCR, grounding, agents	One pretrain stage, then instruct
Swap the LLM	Retrain the projector	Retrain the projector	Retrain the adapter	Not possible later

Step	LLaVA	OneVision	InternVL3
First	Train MLP only, 558k captions	SI: single image, AnyRes	Native pretrain
Second	MLP + LLM, 158k GPT-4 turns	OV: 0.5 / 0.3 / 0.2 mix	Instruction tuning
Third	None	TT: target task mix	None
Frozen	ViT and LLM in stage 1	No weight reset	No freezing stage
Data	LAION-CC-SBU, LLaVA-Instruct	LLaVA-NeXT data plus own	40/35/20/5 text, interleaved, caption, video

Setting	Default	Note
`anyres_cost` tile, thumb	336	576 tokens per tile at patch 14
`square_cost` side	336	Fixed 576 tokens per image
LLaVA prompt image_tokens	576	Replaces the image placeholder
`MRoPEConfig` base	10000.0	RoPE frequency base per band
Qwen2-VL video sampling	1-2 FPS	Accepts any frame count
OneVision planner budget	4096	Visual tokens per sample
Dynamic-FPS demo budget	32768	81 tokens per frame after 3x pool
`dvd_throughput` llm_tokens	128	Output tokens per request

Symptom	Cause	Fix
Squished text, lost chart labels	Square resize to 336	Native aspect ratio with a max-pixels cap
Large image fills the context	AnyRes 4x4 is about 9800 tokens	Native pack or M-RoPE, no tile overhead
Patches attend across images	No block-diagonal mask	Mask from cumulative offsets, or use the `cu_seqlens` path
Weak image skills after training	Video-first or multi-image-first order	Train SI, then OV, then TT
GSM8K drops 5-10 points	Post-hoc alignment debt	Native pretrain with a 40% text floor
Long video runs out of frequency range	Fixed position scale	MRoPE-v2 scaling with max input size

LLaVA you need a cheap, reproducible recipe. Stage 2 takes ~20 hours on 8xA100.
LLaVA-OneVision one model serves single image, multi-image and video at a predictable cost.
Qwen2.5-VL video with timestamps, GUI agents with JSON output, min_pixels and max_pixels caps.
InternVL3 text skills must not regress and you can pay for native pretraining.
Post-hoc recipe you want to swap in a new LLM later. It stays cheaper for most projects.
AnyRes tiling the encoder is frozen at 336x336, but you need detail.

Task	Code
Project ViT patches	`MLPProjector(PATCH_DIM, HIDDEN_DIM, LLM_DIM)`
Build a LLaVA prompt	`build_llava_prompt(system, user, 576)`
Pack a mixed batch	`pack_batch(images, patch)`
Dense block-diagonal mask	`build_dense_mask(pack)`
AnyRes token cost	`anyres_cost(img, tile=336, thumb=336)`
Square-resize cost	`square_cost(img, side=336, patch=14)`
OneVision video plan	`plan_video(budget)`
ViR average tokens	`vir_sim(tiers)`

cfg = MRoPEConfig(hidden=48, temporal_dim=16, height_dim=16, width_dim=16)

mrope_angles(cfg, 0, 5, 7)image patch: t=0, h=row, w=col

mrope_angles(cfg, 3, 5, 7)video frame t=3

mrope_rotate(cfg, vec, 3, 5, 7) # rotates each band by its axis

plan = VideoPlan(duration_s=600.0, tokens_per_frame=81,

budget=32768, motion="low")

plan.fps()first candidate <= fps_max

plan.frame_times()one timestamp per frame

plan.total_tokens()frames times tokens_per_frame

parse_tool_call(raw)json.loads, then slice { to }

LLaVA base: 576 tokens per image. At 2048 context, 1472 tokens stay for text.
AnyRes 2x2: 4 * 576 + 576 = 2880 visual tokens.
AnyRes 4x4: a 1344x1344 image gives 9216 + 576, about 9800 tokens.
Qwen2-VL native: 1120x672 gives 40x24 merged patches, 960 tokens.
OneVision image: AnyRes-9 with 2x2 pool, 9 * 182 + 182 = 1820 tokens.
OneVision video: 32 frames with 3x3 pool, 32 * 81 = 2592 tokens.
Dynamic FPS cap: $inline$
Qwen2.5-VL video: 60 s at 4 FPS with 81 tokens per frame is 19440 tokens.

M-RoPE: Multimodal RoPE. Time, height and width bands in the hidden dim.
V2PE: variable visual position encoding. Learned band split per modality.
NaFlex: SigLIP 2 native-flex mode. One checkpoint serves 256, 729 or 1024 tokens.
ViR: Visual Resolution Router. Picks the minimum resolution before encoding.
DvD: Decoupled Vision-Language deployment. Encoder and LLM on separate GPUs.
SI, OV, TT: OneVision stages. Single-image, OneVision mix, task transfer.

Why did MLP beat Q-Former?: LM loss only and easy LLM swap. Contexts grew to 32k-128k+, so 576 tokens became cheap.
OneVision or Qwen2.5-VL budget?: OneVision fixes the budget and scales the pooling. Qwen2.5-VL budget scales with input.
Why does M-RoPE need no position table?: Each token rotates by its (t, h, w) position. Images use t = 0.
Why not always pretrain native?: Compute is like a text LLM. Interleaved data is rare. The LLM cannot change later.
What does DvD give?: For an 8B LLM with a 400M encoder, about 2x per-node throughput.