Embedding Models — The 2026 Deep Dive
Word2Vec gave you a vector per word. Modern embedding models give you a vector per passage, cross-lingual, with sparse, dense, and multi-vector views, sized to fit your index. Pick wrong and your RAG retrieves the wrong thing. Your RAG system retrieves the wrong passage 40% of the time. The culprit is rarely the vector database or the prompt. It is the embedding model. Choosing an embedding in 2026 means picking across five axes: Dense vs sparse vs multi-vector. One vector per passage, or one per token, or a sparse weighted bag of words. Language coverage. Monolingual English models still win on English-only tasks. Multilingual models win when corpora are mixed. Context length. 512 tokens vs 8,192 vs 32,768 — and real effective capacity is often 60-70% of the advertised max. Dimension budget. 3,072 floats at full precision = 12 KB per vector. At 100M vectors, storage is $1,300/month. Matryoshka truncation cuts this 4×. Open vs hosted. Open-weight means you control the stack and data. Hosted means you trade control for always-latest. This lesson names the tradeoffs so you can pick on evidence, not on whatever was popular last quarter. Dense, sparse, and multi-vector embeddings Dense embeddings. One vector per passage (usually 384-3,072 dimensions). Cosine similarity ranks passages by semantic proximity. OpenAI text-embedding-3-large, BGE-M3 dense mode, Voyage-3. Default choice. Sparse embeddings.…
Embedding Models — The 2026 Deep Dive: Word2Vec gave you a vector per word. Modern embedding models give you a vector per passage, cross-lingual, with sparse,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.