Multimodal Evaluation
Training is half the loop. The other half is measurement. This lesson builds three evaluation surfaces from primitives: image-caption retrieval reported as R@1, R@5, R@10; visual question answering reported as exact match accuracy; and image captioning reported as BLEU-4. Each metric is a function over the model's outputs and a synthetic eval suite that runs in seconds. Compute Recall@K from a similarity matrix between image and caption embeddings. Compute exact-match VQA accuracy from a model that maps (image, question) pairs to a fixed answer vocabulary. Compute BLEU-4 from generated and reference token sequences without any external library. Run all three evals against a synthetic suite built on top of the trained model from lesson 62. The temptation is to declare a multimodal model finished when the training loss plateaus. Training loss measures fit on the training distribution; it does not measure whether the model can rank pairs in a held-out batch, answer a question, or write a caption a human would accept. Three eval surfaces are standard: Retrieval (R@1, R@5, R@10). Build the joint embedding for a query caption; rank every image in the eval pool by cosine; report whether the matching image lands in the top 1, top 5, top 10. Symmetric (image-to-text) form runs the same way. Visual question answering (exact match). Given (image, question), the model outputs…
Multimodal Evaluation: Training is half the loop. The other half is measurement. This lesson builds three evaluation surfaces from primitives: image-caption…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.