Claude Certification Curriculum

RAG, Retrieval, and Data Pipelines

Type: Build

Languages: Python

Prerequisites: End-to-End Architecture and Value Tradeoffs; Phase 11, Lessons 06 and 07; Phase 5, Lesson 23

Time: ~150 minutes

The Problem

A policy assistant works for months. After a document refresh, it begins giving

confident answers based on an old refund threshold. Model version, prompt, and

latency have not changed.

The team adds "use the latest policy" to the prompt. Nothing improves. The

model cannot follow evidence it never received. The index contains both policy

versions, metadata filters are missing, and the retriever ranks the obsolete

chunk first because its wording matches the query more closely.

This is a retrieval incident. Treating it as a model incident wastes time and

can hide the real control failure.

The Concept

RAG Is a Data System

Retrieval-augmented generation has two connected systems with different failure

modes.

flowchart LR
    S["Sources"] --> N["Normalize and classify"]
    N --> C["Chunk with metadata"]
    C --> I["Index and version"]
    Q["User query and identity"] --> R["Retrieve and filter"]
    I --> R
    R --> K["Rank and assemble context"]
    K --> G["Claude generates with citations"]
    G --> V["Validate claims and evidence"]
    V --> O["Answer or escalate"]
    O --> E["Outcome and retrieval eval"]
    E --> R

The model can be excellent while the system fails because:

• the source was never ingested
• parsing dropped the relevant table
• chunking split a condition from its exception
• the index used stale or incompatible representations
• filters ignored tenant, jurisdiction, date, or permission
• ranking favored a keyword match over the authoritative source
• context assembly truncated the best evidence
• generation cited one chunk while claiming more than it supports

Diagnose the earliest failing boundary.

Design Chunks Around Meaning and Retrieval

Fixed token chunks are a baseline, not a universal answer. Chunk shape should

preserve the unit a person would cite.

For prose policies, headings and paragraphs often provide useful boundaries.

For API documentation, keep a method signature with parameters and errors. For

tables, preserve headers with each row group. For tickets, one message may need

conversation context. For source code, functions and classes are better than

arbitrary character windows.

Overlap helps when a fact crosses a boundary, but it also duplicates evidence,

increases index size, and can crowd the final context with near-identical text.

Measure it.

Every chunk needs metadata:

• stable document and chunk identifiers
• source URI or system-of-record identifier
• version and effective date
• tenant, jurisdiction, product, or content type
• access-control attributes
• ingestion and parser version
• parent heading and position

Metadata is how retrieval becomes governed rather than merely similar.

Match Retrieval to Query and Data Shape

#### Sparse Retrieval

BM25-style retrieval matches explicit terms. It is strong for identifiers,

product names, error codes, and policy phrases. It is cheap and explainable.

#### Dense Retrieval

Embeddings match semantic similarity. They help when users paraphrase a concept

or vocabulary differs between query and source. They can miss exact identifiers

and can retrieve semantically related but non-authoritative text.

#### Hybrid Retrieval

Combine sparse and dense candidates, then fuse or rerank. Hybrid retrieval often

handles mixed natural-language and identifier queries better than either alone.

#### Filtered Retrieval

Apply trusted metadata and authorization before evidence reaches the model. Do

not ask Claude to ignore chunks the user is not allowed to see. The forbidden

data should not enter context.

#### Iterative Retrieval

An agent can reformulate queries, follow references, or identify missing

evidence. Use this when discovery is genuinely adaptive. Set query, turn, time,

and cost budgets. Stable question-answering pipelines should not pay agentic

complexity by default.

Separate Retrieval Evaluation From Answer Evaluation

If the correct evidence is absent from the top candidates, answer quality has a

ceiling. Measure retrieval first.

Useful measures include:

• recall at K: did the candidate set contain the required source?
• precision at K: how much of the candidate set was relevant?
• mean reciprocal rank: how early did the first relevant source appear?
• nDCG: did the ranking place highly relevant sources first?
• freshness coverage: did results use the active version?
• authorization leakage: did any result violate the caller's access?

Then evaluate generation:

• claim support by cited evidence
• citation correctness and completeness
• answer completeness
• abstention when evidence is insufficient
• conflict detection across sources

A single end-to-end score cannot tell you which layer to repair.

Preserve Provenance as Data

Do not let provenance exist only as prose generated after the answer. Carry

source identifiers through retrieval, context assembly, output schema, and logs.

For each claim, retain:

• source document and chunk identifier
• source version and effective date
• exact supporting span
• retrieval score and rank
• transformation or summarization steps

If sources conflict, report the conflict. Do not silently choose the most recent

date unless the domain has an explicit precedence rule.

Make Refresh Atomic and Observable

A document refresh can create a mixed index where old and new chunks coexist.

Safer patterns build a new version, validate it, then switch an alias or pointer

atomically. Keep rollback until the new index passes retrieval and freshness

checks.

Monitor:

• ingestion success and lag
• parsed content count and size
• active version by source
• embedding or index version
• empty-result and low-score rates
• retrieval distribution shifts
• top failed evaluation queries

Build It

Interactive Lab

Use the ranking lab to compare lexical matches, metadata filters, stale-source

exclusion, and top-K behavior before editing code. The visible ranks connect a

retrieval decision to recall, reciprocal rank, freshness, and provenance.

Practice Lab

Add a stale or unauthorized document to a copy of the fixture and prove that it

cannot enter the candidate set before generation.

Shipped Artifact

outputs/retrieval-evidence-report.json

is a filled baseline containing ranked chunk identities, active source versions,

and retrieval metrics.

Verify It

Reproduce and verify it with:

bashCopycd certifications/claude/lessons/24-rag-retrieval-and-data-pipelines/code
python3 main.py
python3 -m unittest discover tests -v

The six-question quiz checks diagnosis and retrieval selection.

Capstone Connection

Carry the evidence report into the Architect Professional capstone's RAG

evaluation and freshness gates.

The lab implements a small BM25-style index in the Python standard library. It

is intentionally transparent. Production search systems are faster and more

capable, but the scoring and metadata boundaries should stop feeling magical.

Run it:

bashCopycd certifications/claude/lessons/24-rag-retrieval-and-data-pipelines/code
python3 main.py
python3 -m unittest discover tests -v

Step 1: Normalize Tokens

tokenize lowercases text and extracts alphanumeric terms. Production pipelines

need language-aware tokenization, field handling, and parser tests. The lesson

keeps only the ranking concept.

Step 2: Chunk With Stable Identity

chunk_document creates overlapping word windows while retaining document ID,

position, update time, and a stable chunk ID. Invalid overlap fails early rather

than creating an infinite loop.

Step 3: Exclude Inactive Sources Before Indexing

RetrievalIndex.build ignores inactive document versions. This is a simplified

freshness gate. In production, activation should be tied to a validated index

version and atomic switch.

Step 4: Score Transparently

The index computes term frequency, document frequency, length normalization,

and an inverse-document-frequency score. Exact query terms can raise the source

that actually contains the active policy language.

Step 5: Return Provenance

Every RetrievalHit carries document ID, chunk ID, update date, text, and score.

The generation layer should consume this structured evidence and return claim

links to it.

Step 6: Evaluate the Retriever

evaluate_retrieval calculates recall at K and mean reciprocal rank against

labeled cases. Add normal, ambiguous, stale-version, permission, and adversarial

queries before changing ranking.

Use It

Production systems usually combine a document parser, object storage, sparse or

vector index, metadata filters, reranker, and an evaluation pipeline. Keep the

same contracts even when managed services hide the implementation.

For the policy incident:

1. Reproduce the query and inspect the retrieved chunk IDs.
2. Confirm which source versions are active in the index.
3. Check parsing and chunk boundaries around the threshold and exception.
4. Verify identity and metadata filters.
5. Compare sparse, dense, and hybrid candidate sets.
6. Run the frozen retrieval evaluation before and after the repair.
7. Switch the validated index atomically and retain rollback.
8. Re-run claim-support evaluation at the generation layer.

Do not start by changing temperature or model size. Neither can recover missing

or forbidden evidence.

Exam Decision Patterns

When answers became wrong immediately after a document refresh while model and

latency stayed stable, investigate ingestion, indexing, filtering, and retrieval

first.

Strong architecture choices:

• match retrieval to exact identifiers and semantic paraphrases
• filter by identity and metadata before generation
• version sources and indexes
• evaluate retrieval separately from final answers
• carry provenance through the output contract
• represent insufficient or conflicting evidence explicitly

Weak choices:

• tell the model to remember the latest document
• increase context with every source
• replace the model before inspecting candidates
• rely on generated citations without source identifiers

Common Traps

More Context Means More Grounding

Irrelevant context competes for attention and can hide the best evidence. Better

retrieval and ordering often beat a larger context payload.

Similar Means Authoritative

Semantic similarity does not encode policy precedence, permissions, or effective

date. Those need metadata and rules.

Valid Citation Means Supported Claim

A citation can point to a real source that does not support the whole claim.

Evaluate entailment and coverage, not only link validity.

Refresh Means Append

Appending new chunks without deactivating old versions creates contradictory

evidence. Treat refresh as a versioned deployment.

Exercises

1. Add field-aware boosting so title matches score more than body matches.
2. Add a jurisdiction filter and a test proving unauthorized chunks never

appear in candidates.

1. Build a hybrid rank-fusion function over two ranked lists.
2. Create ten retrieval cases where exact identifiers and paraphrases require

different strategies.

1. Design an atomic index-refresh checklist with validation and rollback.

Key Terms

Further Reading

• Claude citations documentation for current citation support
• Claude token counting documentation for context budgeting
• Phase 11, Lesson 06 for a RAG pipeline from first principles
• Phase 11, Lesson 07 for advanced retrieval and reranking
• Phase 19, Lesson 65 for hybrid sparse and dense retrieval

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.