InternVL3: Native Multimodal Pretraining
Every open VLM before InternVL3 followed the same three-step recipe: take a text LLM trained on trillions of text tokens, bolt on a vision encoder, then fine-tune the seams. This works but has alignment debt — the text LLM has spent its full pretraining budget on pure text and does not natively understand visual tokens. When you add vision post-hoc, the LLM has to re-learn how to relate visual input to its text reasoning without forgetting the text. InternVL3 (Zhu et al., April 2025) rejects the post-hoc approach: one pretraining run, text and multimodal interleaved from step one. The result matches Gemini 2.5 Pro on MMMU-Pro at 78B params open. This lesson reads the case for native pretraining and what changes when you make it. Explain why post-hoc VLM training accumulates alignment debt, citing the three measurable symptoms (catastrophic forgetting, answer drift, visual-text inconsistency). Describe InternVL3's native pretraining corpus mix and why the ratio of text : interleaved : caption matters. Compare V2PE (variable visual position encoding) to Qwen2-VL's M-RoPE. Name the Visual Resolution Router (ViR) and Decoupled Vision-Language (DvD) deployment optimizations. Post-hoc VLM training is the default. LLaVA, BLIP-2, Qwen-VL, Idefics — all take an already-pretrained LLM (Llama, Vicuna, Qwen, Mistral) and add vision. The training stages typically look like: Frozen LLM + frozen vision encoder + trainable projector,…
InternVL3: Native Multimodal Pretraining: Every open VLM before InternVL3 followed the same three-step recipe: take a text LLM trained on trillions of text…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.