Cheat sheets

Retrieval-Augmented Generation (RAG)

Chunk, index, retrieve, fuse, rerank, generate with citations and grade each stage. Python from the lessons.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

End-to-end RAG pipeline#

chunksrewritessearchtop_ntop_kDocumentsoffline indexingChunkerrecursive splitHybridIndexBM25 + denseGeneratorcited answer orrefuseUser queryRewriterHyDE, multi-query,decomposeRetrieverRRF, top_n 12Rerankercross-encoder,top_k 5chunksrewritessearchtop_ntop_kDocumentsoffline indexingChunkerrecursive splitHybridIndexBM25 + denseGeneratorcited answer orrefuseUser queryRewriterHyDE, multi-query,decomposeRetrieverRRF, top_n 12Rerankercross-encoder, top_k5

From Lesson 19.69, 19.69 code/main.py

One query: Pipeline.query#

1Rewriterpick_strategy picks decompose for " and ", hydefor api or config, else multiquery.2HyDEBM25 searches the query. Dense searches thehypothetical passage.3Subquerieseach rewrite searches alone. rrf fuses theranked lists.4HybridIndexBM25 top 8 and dense top 8, fused, keep top_n12.5Rerankercross-encoder score plus 0.3 times queryoverlap. Keep top_k 5.6Generatortop score below REFUSE_THRESHOLD gives "I do notknow."7Generatorelse one sentence from up to 2 chunks, each witha [doc_id:chunk_index] anchor.

From 19.69 code/main.py, Lesson 19.69

Tasks in code#

TaskCode
Fixed window chunksfixed_window(doc_id, text, size=400, overlap=80)
Chunk at headingsstructural_markdown(doc_id, text)
Score a chunkereval_recall(chunker, fixture, ks=(1, 3, 5))
Fuse ranked listsrrf(rankings, k=60, weights=[1.0, 1.0])
Retrieve through a rewriterretrieve_with_rewriter(query, rewriter, retriever)
Rerank candidatesrerank(model, query, candidates, top_k)
Graded ranking scorendcg_at_k(retrieved, graded, k)
Run the gated demopython3 code/main.py exits 1 on a FAIL

From 19.64 code/main.py, 19.65 code/main.py, 19.67 code/main.py, 19.68 code/main.py, 19.69 code/main.py

Hybrid retrieval with RRF#

retriever = HybridRetriever()BM25Index + DenseIndex
for d in CORPUS:
retriever.add(d)indexes both modalities
result = retriever.search(q, k_each=5, k_out=5)
result["bm25"], result["dense"], result["fused"]three ranked lists
bm25 = BM25Index(k1=1.5, b=0.75)title tokens count 3 times
idf = math.log((n - df + 0.5) / (df + 0.5) + 1.0)stays positive
score[doc.doc_id] += w * (1.0 / (k + rank + 1))rank starts at 0

From 19.65 code/main.py

Assemble and query the pipeline#

p = Pipeline()chunker, index, rewriter, reranker
p.ingest(CORPUS)recursive split, target 400
p.train_reranker_on(TRAIN_TRIPLES)fits the small cross-encoder
result = p.query("<question>")
result.answersentences plus anchors
result.citationslist of doc_id:chunk_index
result.rewriter_strategyhyde, multiquery or decompose
result.latency_msrewrite, retrieve, rerank, generate

From 19.69 code/main.py

Defaults and numbers#

SettingDefaultNote
BM25 k11.5Lesson 11.07 code uses 1.2
BM25 b0.750 ignores length, 1 full normalization
RRF k60Rank 1 adds 1/61
Chunk size256-512Tokens, with 50-token overlap
Chunks per query5 to 102,500-5,000 tokens of context
Rerank pool N50 to 200Cross-encoder keeps the top K
top_n, top_k12Pipeline keeps 5 after rerank
REFUSE_THRESHOLD0.05Below it the answer is "I do not know."

From 19.65 code/main.py, Lesson 19.65, Lesson 11.06, Lesson 11.07, Lesson 19.66, 19.69 code/main.py

Chunking strategies#

CommandWhat it does
recursive_splitProse without structure. Strongest single baseline
structural_markdownMarkdown, RFC and API docs. Cuts at headings
sentence_chunksLong single-topic paragraphs. Target 500
semantic_chunksShort mixed-topic paragraphs. Threshold 0.6
fixed_windowA control only. Cuts mid-symbol in code
Parent-documentSearch small children, return the parent. Dedupe parents
Contextual retrievalPrepend 50-100 words of LLM context to each chunk
Late chunkingEmbed the whole doc per token, then pool per chunk

From Lesson 19.64, 19.64 code/main.py, Lesson 05.23

Key formulas#

Cosine
\cos(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}, from -1 to 1.
BM25 term
\mathrm{IDF}(t) \frac{tf \, (k_1 + 1)}{tf + k_1 (1 - b + b \, |d| / \mathrm{avgdl})}
RRF
\mathrm{RRF}(d) = \sum_R \frac{1}{k + \mathrm{rank}_R(d)}
Precision@k
gold docs in the top k, divided by k.
Recall@k
gold docs in the top k, divided by all gold docs.
MRR
mean of 1 / rank of the first relevant doc.
DCG gain
\frac{2^{rel} - 1}{\log_2(i + 2)} for position i from 0.
nDCG@k
DCG divided by the DCG of the ideal order. From 0 to 1.

From Lesson 11.06, Lesson 11.07, Lesson 19.68, 19.68 code/main.py

Troubleshooting#

SymptomCauseFix
Low recall@k and precision@kChunk cut the answer, or retriever misses itFix boundaries or add a modality
Good recall@k, low MRRRight chunk not at position 1Add a cross-encoder reranker
High MRR, low faithfulnessGenerator invents contentForce cite or refuse in the prompt
High faithfulness, low relevanceGrounded but off-topic answerRewrite the query
Exact code E-4021 missedEmbedding treats it as noiseAdd BM25 and fuse with RRF
Rerank shows no liftN equals KSet N to at least 3 times K
HyDE drops the right docInvented identifier in the passageCap length, weight BM25 lower
40-token fragmentsSemantic chunks without a floorEnforce min_tokens

From Lesson 19.68, Lesson 11.07, Lesson 19.66, Lesson 19.67, Lesson 05.23

Abbreviations#

RAG
Retrieval-Augmented Generation. Retrieve, put in the prompt, generate.
BM25
Best Matching 25. Keyword ranking with IDF and length normalization.
RRF
Reciprocal Rank Fusion. Sums 1/(k + rank) over ranked lists.
HyDE
Hypothetical Document Embeddings. Search with a fake answer.
MRR
Mean Reciprocal Rank. Mean of 1/rank of the first hit.
nDCG
Normalized Discounted Cumulative Gain. Graded ranking quality.
Qrels
gold labels. Queries with gold docs and answers.
Bi / cross-encoder
embeds query and doc apart, or reads them together.

From Lesson 11.06, Lesson 11.07, Lesson 19.66, Lesson 19.68

Questions#

Why RRF, not score mixing?
BM25 scores are unbounded. Ranks compare across modalities.
Why rerank only the top N?
A cross-encoder is 100-1000x slower than a bi-encoder.
Why quote recall@k?
The generator can drop a bad chunk. It cannot use a chunk it never saw.
Why does a wrong HyDE answer help?
The retriever uses its words, not its facts.
RAG or fine-tuning?
RAG for facts. Fine-tuning for style, tone or a reasoning pattern.

From Lesson 19.65, Lesson 11.07, Lesson 19.68, Lesson 19.67, Lesson 11.06

Tips#

  • Start with recursive 512 tokens. Measure recall@5 on a 50-query eval set.
  • Tell the model to use only the context and to say "I don't know."
  • Pin the bi-encoder, cross-encoder and N together.
  • Judge with a different model family than the generator.
  • Slice qrels by query class: literal, paraphrased, multi-topic.
  • Tier the eval: 20 queries in CI, 200 nightly, 2000 weekly.

From Lesson 05.23, Lesson 05.13, Lesson 19.66, Lesson 19.68