Cheat sheets

Production RAG Chatbots and Freshness

Filter, fuse, cache, guard and cite. Keep the index fresh and grade every ranking change.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

One regulated chat turn#

okeligibletop 3sha256 prefixanswerclientQueryrole +jurisdictionprocessInput guardBLOCKED_PATTERNSprocessAccess filterrole or public,tag or anystoreHybridretrievedense + BM25,RRF 60processPromptLayoutsystem, policy,contextstorePromptCachecache_key hit ormissmodelSynthesizecite or sayunsureprocessPII scrubemail and ssnredactedokeligibletop 3sha256 prefixanswerclientQueryrole + jurisdictionprocessInput guardBLOCKED_PATTERNSprocessAccess filterrole or public, tag oranystoreHybrid retrievedense + BM25, RRF 60processPromptLayoutsystem, policy,contextstorePromptCachecache_key hit or missmodelSynthesizecite or say unsureprocessPII scrubemail and ssn redacted

From 19.08 code/main.py, Lesson 19.08

Run, grade and inspect#

CommandWhat it does
python3 scripts/project_test.py <project> --init learning-artifacts/<project>Makes your workspace. A fresh starter fails
python3 scripts/project_test.py <project> --stage 1 --path learning-artifacts/<project>Grades one stage of your code
python3 scripts/project_test.py <project> --all --solution --strictChecks the reference solution
python3 cli.py samples/docs "When does cache expire?" --output answer.json --html answer.htmlWrites a citation receipt as JSON and HTML
cli.py query orchard-index.json tokens --now 210 --max-age 60Queries the freshness snapshot
cli.py <notes> "release replicas" --aliases <aliases.json> --out matches.jsonSearches a markdown folder
cli.py <baseline> <candidate> <judgments> --out retrieval-diff.jsonDiffs two ranking runs per query
python main.pyRuns a capstone scaffold offline

From doc-qa-with-citations/README.md, rag-freshness-pipeline/README.md, semantic-notes-search/README.md, retrieval-evaluation-lab/README.md, 19.02 code/main.py, 19.02 code/ts/README.md

Cache-aware chat turn#

ok, reason = llama_guard_input(query)regex block list first
hits = retrieve(query, role, jurisdiction, corpus, k=3)
context = [f"[{c.anchor()}] {c.text}" for c, _ in hits]
layout = PromptLayout(system=SYSTEM_PROMPT,very stable
policy=f"role={role} jurisdiction={jurisdiction}",
context=context, question=query)question is not cached
cache_hit = cache.check(layout.cache_key())sha256 of prefix, 16 hex
answer = presidio_scrub(answer)redacts [email] and [ssn]
cache.hit_rate()hits / (hits + misses)

From 19.08 code/main.py

Codebase hybrid retrieval#

tokens = (tokenize(chunk.symbol) * 4 +symbol weight 4
tokenize(chunk.summary) * 2 +summary weight 2
tokenize(chunk.body))body weight 1
dense_hits = dense.search(query, k=10)
sparse_hits = bm25.search(query, k=10)
fused = rrf(dense_hits, sparse_hits)k_rrf: int = 60
top = rerank(query, fused, top_k=5)0.3 symbol, 0.1 summary
citations = [c.anchor() for c, _ in top]repo/path:start-end

From 19.02 code/main.py

Project inputs, outputs and checks#

FactorDoc QAFreshnessNotes searchEval lab
BuildsCitation receiptFresh indexLexical TF-IDF searchRegression diff
InputFolder of .txtFull corpus JSONFolder of .md3 JSON files
Outputanswer.json, answer.htmlSnapshot, changesmatches.jsonretrieval-diff.json
Key fieldsstart, end, sha256version, age, hashscore, matcheddelta_ndcg
Statesanswered, abstainedinsert to deletetop k matchesregressions count
Guardneeds_reviewstale version errork must be positive--fail-on-regression

From doc-qa-with-citations/README.md, solution/cli.py, rag-freshness-pipeline/README.md, solution/cli.py, semantic-notes-search/README.md, solution/cli.py, retrieval-evaluation-lab/README.md, solution/cli.py

Project functions#

TaskCode
Load text files with SHA-256load_documents(root)
Chunk with offsetschunk_document(doc, size=200, overlap=30)
Gate a model quoteanswer(question, chunks, model)
Plan index changesdiff(previous, incoming)
Commit a snapshotcommit(path, documents, expected_version)
Drop expired evidenceretrieve(documents, query, now, max_age=3600, k=3)
Build a TF-IDF indexbuild_index(documents, aliases=None)
Score two systemscompare_systems(systems, judgments, k)

From doc-qa-with-citations/API.md, doc-qa-with-citations/solution, rag-freshness-pipeline/solution, solution/main.py, solution/main.py

Freshness ingest and query#

1normalizeNFC text, CRLF to LF, strip, then SHA-256 of thebody.2diffan id that is new goes to insert.3diffa changed hash goes to update. Only a newupdated time goes to refresh.4diffan id that is not in the full corpus goes todelete.5commitflock, compare expected_version, write a tempfile, fsync, os.replace.6retrieveskip a doc when age is below 0 or above max_age.7retrievescore is matched terms over query terms. Sort byscore, age, id.

From rag-freshness-pipeline/solution, rag-freshness-pipeline/README.md

Defaults and numbers#

SettingDefaultNote
Doc QA chunk size200overlap is 30
--max-age3600Older evidence is skipped
--k3Freshness, notes and eval CLIs
Note size limit100 KBLarger notes raise an error
Relevance grades0..3Integers only
Rerank pooltop-20Top-5 go to synth
Cache hit target60-80%Stable prefixes first
Drift alert5%Weekly retrieval quality drop

From doc-qa-with-citations/API.md, solution/cli.py, solution/cli.py, solution/main.py, Lesson 19.08, Lesson 19.02

Troubleshooting#

SymptomCauseFix
"unsupported model answer"Quote is not in the cited chunkReturn an exact span and its chunk id
state is needs_reviewQuote has no query termJudge relevance by hand
state is abstainedRetrieval found no chunksExpected for an off-topic query
"stale index version"Another writer committed firstRead the snapshot, pass its version
"duplicate document id"Two incoming docs share an idMake ids unique
Edited policy still citedIndex lags the sourceIngest the full corpus again
"symlink notes are not supported"Note is a linkUse a regular file in the folder
Generated code poisons hitsBoilerplate in the corpusFlag it "generated" and down-weight

From doc-qa-with-citations/solution, doc-qa-with-citations/README.md, rag-freshness-pipeline/solution, rag-freshness-pipeline/README.md, solution/cli.py, Lesson 19.08, Lesson 19.02

Ranking metrics in code#

Precision@k
relevant hits in the top k, divided by k.
Recall@k
relevant hits divided by all docs with a grade above 0.
RR
1 over the rank of the first relevant doc, else 0.
nDCG
DCG with gain 2 ** grade - 1 over the ideal DCG.
Unjudged
docs in the top k with no label. They get zero gain.
Notes IDF
math.log((1 + len(terms)) / (1 + count)) + 1.
Doc QA score
sum of (1 + math.log(bag[t])) times an IDF term.

From solution/main.py, solution/main.py, solution/retrieval.py

Questions#

Why filter by role first?
The access policy must hold before any score.
Why put the question last?
The system prompt and context form the cached prefix.
What does a re-index touch?
Only chunks whose text changed. Edges change when imports change.
Does a valid citation prove an answer?
No. It can still be irrelevant or outdated.
Is a refresh a rewrite?
No. Same content with a later time only updates the time.

From Lesson 19.08, Lesson 19.02, doc-qa-with-citations/README.md, rag-freshness-pipeline/README.md

Tips#

  • Label 100 cross-repo questions with gold file:line answers.
  • Keep a 200 Q/A golden set and 50 adversarial prompts.
  • Reject any claim without a (repo/path:start-end) anchor.
  • Re-index a 50-file push in under 60 seconds.

From Lesson 19.08, Lesson 19.02