Vision-Language Pretraining
The encoder, projection, and decoder are wired. Now train them together. Two objectives drive learning: a contrastive image-text loss (InfoNCE) that pulls matching pairs together in the joint embedding space, and a language modeling loss that asks the decoder to caption each image. Combined, they teach the network both to find the right image for a caption and to write a caption for the image. Implement InfoNCE contrastive loss across a batch of image-caption pairs. Compose contrastive loss with autoregressive language modeling loss. Synthesize a 200-pair mock image-caption corpus with no real dataset download. Run a 50-step demo training loop and observe both losses decreasing. A vision-language model needs two skills. It must rank: given a caption, find the right image among many. It must generate: given an image, write a caption. Pretraining the model on one skill alone gives you half a system. CLIP nailed ranking but cannot caption. GPT-4V can caption but uses a separate retrieval head for ranking. Multi-objective pretraining gets both in one pass. InfoNCE handles the ranking half. For a batch of N pairs, the model treats the N matching pairs as positives and the N^2 - N mismatched pairs as negatives, then runs a cross-entropy loss on the resulting (N, N) similarity matrix. The LM loss handles the generation half: standard next-token prediction conditioned…
Vision-Language Pretraining: The encoder, projection, and decoder are wired. Now train them together. Two objectives drive learning: a contrastive image-text…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.