Applications

RAG

Chunking, hybrid retrieval, reranking and how to measure a retrieval pipeline.

26 of 15 planned

RolesLLM Product EngineeringAI Evaluation and ReliabilityAI Data SystemsCustomer AI DeploymentDeveloper Experience and Education

Level

26 items

  1. Retrieval or fine-tuning for the policy wikiL1ScreenConcept6 min
  2. The first metric to read after a wrong answerL1ScreenConcept6 min
  3. The retriever model and the reranker modelL1ScreenConcept6 min
  4. Why vector search ranks by cosineL1ScreenConcept5 min
  5. Count the work of a reranker over the whole indexL2WorkingEstimate8 min
  6. Faithfulness and answer relevance grade different thingsL2WorkingConcept7 min
  7. Filter on metadata ahead of the vector searchL2WorkingDesign8 min
  8. Read the parent-document retrieval functionL2WorkingCode reading8 min
  9. Read the rank fusion functionL2WorkingCode reading8 min
  10. Review a chunking choice before the first index buildL2WorkingTrade-off10 min
  11. Settle a chunk overlap argument with one ablationL2WorkingTrade-off8 min
  12. The reranker is perfect offline and worse in productionL2WorkingDebug8 min
  13. Vector search misses the article with the exact error codeL2WorkingDebug8 min
  14. Which query rewriter fits a two-topic questionL2WorkingTrade-off9 min
  15. Why hybrid search fuses ranks and not scoresL2WorkingConcept8 min
  16. Build the eval that picks a retrieval configurationL3SeniorTake-home60 min
  17. Design a question answering service over company documentsL3SeniorDesign30 min
  18. Faithfulness stays high while reviewers find invented factsL3SeniorDebug10 min
  19. Let the assistant say it does not knowL3SeniorDesign10 min
  20. Pick how many candidates the reranker scoresL3SeniorTrade-off10 min
  21. Read the four lines that score a BM25 termL3SeniorCode reading10 min
  22. Recall holds, MRR drops, nobody touched the retrieverL3SeniorDebug10 min
  23. The invented passage made code search worseL3SeniorDebug10 min
  24. Design code search across ten repositories that stays freshL4StaffDesign30 min
  25. Every metric is high and one unit still gets wrong answersL4StaffDesign15 min
  26. The other jurisdiction's clause reached the answerL4StaffDesign15 min

Taught in

The answers on this card come from these lessons.