Count the work of a reranker over the whole index
RolesLLM Product EngineeringAI Data SystemsCustomer AI Deployment
The question
A proposal replaces the vector retriever with a cross-encoder that scores every chunk in a 10 million chunk index, so that nothing is ever missed. Estimate the work for one question, and say what you propose instead.
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
One forward pass per chunk means 10 million forward passes for one question, which no request budget allows. Keep a bi-encoder for the first stage, because its document vectors are computed once at index time, and run the cross-encoder on a pool of 50 to 200 candidates. The advanced RAG lesson puts the per-pair cost at 100 to 1000 times a bi-encoder comparison.
Show full answer
The count
| Stage | Work at index time | Work per query |
|---|---|---|
| Bi-encoder | one embedding per chunk, stored | one embedding of the query, then one vector comparison per chunk |
| Cross-encoder | nothing can be stored, the score depends on the query | one forward pass per pair, so one per chunk in the pool |
With the cross-encoder over the whole index, the pool is the index. An index of 10 million chunks means 10 million passes for one question.
The staged shape
Retrieve a larger candidate set with the cheap stage, for example the top 50 from hybrid search, and rerank with the cross-encoder to the final top 5. The expensive model runs 50 times instead of 10 million, and its lift lands where it matters, in the final order.
Two more numbers to plan
- A production cross-encoder of about 22M parameters takes 88 MB at float32. Load it once and keep it resident.
- Run the N candidates as one batch. Scored one at a time, the latency multiplies by N.
Follow-ups
L2WorkingWhich part of the bi-encoder cost is paid once?
Show expected answer
the document embeddings, at index time; a query pays for one embedding and the comparisons.
L2WorkingWhy does the expensive model pay for itself on the top 50?
Show expected answer
50 passes fit a request budget and the lift is concentrated in the final order of the candidates.
L3SeniorThe index doubles in size. Which stage grows in cost per query?
Show expected answer
the first stage, because it compares against every chunk; the reranker stays at N passes while N is fixed.
L3SeniorAt this scale the first stage itself is not a brute-force scan. What does it use, and what does that cost?
Show expected answer
an approximate nearest neighbor index, which trades a little recall for speed; the advanced RAG lesson notes that at 2 million chunks the error can push a relevant chunk out of the top k.
Wrong answers
Show the wrong answers that interviewers hear
“Score everything, GPUs are fast.”
10 million passes per question is not a latency a user waits for, and the lesson's model takes 88 MB before any batch.
“Precompute the cross-encoder score for every chunk.”
The score depends on the query, so there is nothing to precompute.
“Use a smaller cross-encoder over the whole corpus.”
Smaller rerankers lose quality faster than they save latency, and the pass count stays the same.
Taught in
The lesson cross-encoder counts the passes and sizes the model, and the lesson advanced RAG gives the cost gap and the top 50 to top 5 funnel.