Hybrid Retrieval with BM25 and Dense Embeddings
Lexical and semantic retrieval fail on opposite query distributions. Hybrid retrieval with reciprocal rank fusion does not interpolate, it votes - and the vote wins on every query class. Implement BM25 from scratch from the Robertson and Sparck Jones formulation, with field weighting, document length normalization, and tunable k1 and b. Build a dense retriever on top of a deterministic mock embedding so the loop runs offline. Implement reciprocal rank fusion exactly as Cormack, Clarke, and Buettcher published it in 2009, and explain why it dominates score-weighted interpolation. Tune the RRF k constant and the per-modality weights and read the trade-offs on a small fixture corpus. Lexical search wins when the query carries a literal identifier the corpus contains verbatim. A query for AbortMultipartOnFail returns the right Go function via BM25 in microseconds. The same query, embedded, sits at the boundary of three similarity clusters and a dense retriever ranks the wrong file first. Dense search wins when the query is paraphrased away from the corpus's literal tokens. A user asking "how do we handle cancelled uploads" never typed the word abort or multipart. BM25 returns the documentation chunk on "uploading large files" because that page contains the word uploads. Dense retrieval finds the abort function whose summary mentions cancellation. The choice between the two is not a static one.…
Hybrid Retrieval with BM25 and Dense Embeddings: Lexical and semantic retrieval fail on opposite query distributions. Hybrid retrieval with reciprocal rank…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.