Vision-Language Model Architectures

ViT, bridge, LLM. Projector, Q-Former, Perceiver, gated cross-attention, training stages and recipe numbers.

The ViT-MLP-LLM pattern#

MLP projector: Linear, GELU, Linear. Maps (N_patches, d_vit) to (N_patches, d_llm).
Q-Former: 32 learnable queries cross-attend to frozen ViT patches. Output is 32 visual tokens.
Perceiver resampler: K = 64 learnable latents. Any patch count exits as 64 tokens.
Gated cross-attention: $inline$ . Alpha starts at 0.
Cross-attention: Q comes from the text stream. K and V come from the image stream.
Cosine alignment loss: $inline$ . 0 is aligned, 2 is opposite.
DeepStack: stacks features from several ViT depths, not the last layer only.
CMER: share of outputs with high text confidence and low image-text similarity.

Task	Code
Build a projector	`Projector(vit_dim=768, llm_dim=4096, hidden=4096)`
Stack ViT depths	`deepstack_features(per_layer_features)`
Measure CMER	`cross_modal_error_rate(image_emb, text_emb, text_confidence)`
Resample patches to fixed latents	`perceiver_resampler(patches, num_latents=8, num_blocks=2)`
Gated step, closed gate	`gated_cross_attention_step(text_hidden, visual, alpha=0.0)`
One image per text token	`interleaved_mask(seq)`
LLaVA prompt with 576 tokens	`build_llava_prompt(system, user, image_tokens=576)`
Self-attention mask	`causal_mask(length)`

vision_tokens = self.vit(image)(B, N_patches, d_vit)

vision_embeds = self.projector(vision_tokens)(B, N_patches, d_llm)

text_embeds = self.llm.get_input_embeddings()(input_ids)(B, M, d_llm)

out = text_embeds.clone()copy, then fill

for b in range(input_ids.size(0)):one sample at a time

positions = (input_ids[b] == self.image_token_id).nonzero(as_tuple=True)[0]

out[b, positions] = vision_embeds[b]<image> slots get vision

cfg = AlignConfig()768 -> 1024 -> 512, 200 steps

encoder = VisionEncoder(encoder_cfg).eval()

freeze(encoder)requires_grad_(False)

projector = MLPProjector(cfg.vision_hidden, cfg.projection_hidden, cfg.text_hidden)

with torch.no_grad():

_, cls = encoder(img)CLS pool, no encoder grads

text_emb = text(ids)frozen MockTextEmbedding

image_emb = projector(cls)

loss = cosine_alignment_loss(image_emb, text_emb)

Symptom	Cause	Fix
`ValueError` on image tokens	Placeholder count differs from patches	Pre-pad each sample to the same count
Confident wrong objects	12% of crawled pairs are not grounded	Track CMER, route high CMER to review
Above, below, count wrong	VLMs score 50-60% on spatial tests	Use a pose, depth or detection model
Text skill drops after tuning	The gate got too wide too fast	Start alpha at 0, check the gate schedule
Weak OCR	Too few visual tokens	Use 576-1024 tokens or AnyRes
Context fills fast	576 tokens per image, more per frame	Use a Q-Former at 32 tokens

VLM: Vision-Language Model. Image encoder plus a full language model.
ITC: image-text contrastive. CLIP-style loss on pooled queries and text CLS.
ITM: image-text matching. Binary classifier on hard negatives.
ITG: image-grounded text generation. Causal LM loss on the queries.
CMER: Cross-Modal Error Rate. The hallucination KPI.
AnyRes: split a high-res image into tiles and concatenate their tokens.

Q-Former or MLP?: Q-Former when token budget matters. MLP when quality per token matters.
BLIP-2 or Flamingo?: BLIP-2 for single-image VQA on a budget. Flamingo for few-shot and many images.
Why init alpha at 0?: At step 0 the model is the pretrained LLM on text.
Why no mask on cross-attention?: The whole image is visible before any text.
Why two layers, not one?: GELU gives one non-linear bend. One linear layer only rotates and scales.
What to ablate first?: Visual-token count explains ~60% of variance. Encoder ~20%. Connector ~5%.

Encoder: SigLIP 2 SO400m/14. Add DINOv2 ViT-g/14 for segmentation or grounding.
Connector: 2-layer MLP on patch tokens. Skip Q-Former unless you are token-constrained.
LLM: 7B for cost, 70B for quality. LLM size sets the ceiling.
Data: PixMo + ShareGPT4V + Cauldron. Dense human captions beat distillation.
Train Stage 1 at 384 fixed. Ramp to dynamic resolution up to 1280 for OCR.
Swap the encoder before the connector. SigLIP over CLIP added 3+ points MMMU.