Projection Layer for Modality Alignment
A vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text embedding space, and a cosine alignment loss against a paired caption pulls the two spaces into agreement. That projection is the smallest piece of a vision-language model and the one that matters most for transfer. Build a two-layer MLP projection that maps image features into the text embedding space. Construct a mock text embedding table (no pretrained tokenizer, no real corpus). Compute a cosine alignment loss between projected image tokens and a paired caption embedding. Train the projection alone with a frozen vision encoder and a frozen text table. You have a vision encoder (lessons 58-59) producing tokens of dimension visionhidden = 768. You have a text decoder you want to bolt on top with embedding dimension texthidden = 512 (any other number is just as plausible). The decoder expects text-shaped tokens. The image tokens are not text-shaped: they live in a basis the encoder learned during vision-only pretraining, with no relationship to the decoder's word vectors. Two-layer MLP projection (linear, GELU, linear) bridges the gap. It is small enough (about 768 1024 + 1024 512 = 1.3M parameters) to train in minutes on a single GPU, and it is the…
Projection Layer for Modality Alignment: A vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces.…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.