Claude Certification Curriculum

Reliable Extraction, Batch, and Independent Reviewers

Valid JSON proves the shape survived. It does not prove the facts did.

Type: Reference

Languages: Python

Prerequisites: Validate the Claim, Not the Confidence, Structured Output Is an Untrusted Contract; Phase 14, Lesson 39

Time: ~135 minutes

🎯 Learning Objectives

Define extraction criteria that reduce false positives and ambiguous labels
Use schemas, examples, nullable fields, enums, and evidence spans deliberately
Separate syntax, schema, semantic, and provenance validation
Design bounded retry and independent reviewer passes
Choose real-time or batch processing from workflow requirements

The Problem

A pipeline extracts contract obligations into valid JSON. Every record matches

the schema. Legal reviewers still reject 18 percent.

The model fills missing dates with plausible values, labels background statements

as obligations, and maps unfamiliar categories to the nearest enum. A retry loop

feeds the same prompt back until validation passes. Since validation checks only

types, the invented values become more confidently formatted.

The team solved serialization and mistook it for correctness.

The Concept

Define the Judgment Before the Schema

A schema says what fields exist. Criteria say what qualifies.

For an obligation extractor, define:

obligated party must be explicit or unambiguously linked
required action must be stated, not merely discussed
trigger and deadline are extracted only when supported
evidence span must contain the claim
unknown values remain null
unsupported category uses other with a note or triggers review
exceptions and negations change the result

Without these rules, annotators, model, and evaluator apply different tasks.

Use Few-Shot Examples for Boundaries

Examples are most useful where reasonable people make different judgments.

Include:

a clear positive
a near miss
a negated obligation
missing date represented as null
a category outside the enum
two obligations in one paragraph
conflicting clauses

Each example should demonstrate the reason, not only the answer. Do not flood

context with redundant easy cases.

Make Absence Representable

If a field may be unknown, the schema needs an explicit state. Forcing a string

encourages invention.

json{
  "type": "object",
  "properties": {
    "party": {"type": ["string", "null"]},
    "action": {"type": "string"},
    "deadline": {"type": ["string", "null"]},
    "category": {
      "type": "string",
      "enum": ["payment", "delivery", "reporting", "other"]
    },
    "evidence_span": {"type": "string"},
    "needs_review": {"type": "boolean"}
  },
  "required": ["party", "action", "deadline", "category", "evidence_span", "needs_review"],
  "additionalProperties": false
}

Required plus nullable forces an explicit decision: supported value or known

absence. It prevents silent field omission.

Use Tool Use for Typed Output

A no-side-effect extraction tool can carry the schema. Tool choice can require

that typed record when the application needs it. Strict schema features can

guarantee valid structure where current APIs support them.

Do not call a real action tool just to obtain structured output. Extraction and

execution have different authority.

Validate in Four Layers

flowchart LR
    O["Model output"] --> J{"Syntax valid?"}
    J --> S{"Schema valid?"}
    S --> M{"Semantics valid?"}
    M --> P{"Evidence supports claim?"}
    P --> A["Accept"]
    J -->|"no"| R["Targeted repair"]
    S -->|"no"| R
    M -->|"no"| H["Retry or review"]
    P -->|"no"| H

#### Syntax

Can the payload be parsed?

#### Schema

Are fields, types, enums, and bounds valid?

#### Semantics

Do cross-field relationships hold? A deadline cannot precede an effective date

when the domain forbids it. A needs_review false result cannot accompany an

unsupported category.

#### Provenance

Does the evidence span actually support the extracted claim, and does it come

from the correct source version?

Only the last two detect many confident hallucinations.

Feed Back the Smallest Useful Error

On repair, return structured validation feedback:

json{
  "category": "semantic_validation",
  "field": "deadline",
  "message": "The extracted date does not appear in the evidence span.",
  "allowed_action": "Set deadline to null or select a supported span."
}

Do not say only "try again." Keep the original source and prior result. Limit

retries. Repeated semantic failure should escalate instead of converting

uncertainty into latency and cost.

Separate Generator and Reviewer

The generator extracts. The reviewer receives source, candidate record, and a

rubric. It checks:

required evidence exists
span supports every non-null claim
negation and exceptions were handled
category fits the definition
unknowns were not invented
conflicts and ambiguity are flagged

Use a fresh context for stronger independence. The reviewer returns finding IDs,

fields, evidence, and disposition. It does not silently rewrite the record.

Measure reviewer precision and recall against human labels. A model judge is an

instrument, not ground truth.

Choose Batch for the Workflow

The July 2026 CCAR-F public guide specifies a 50 percent Message Batches cost

reduction, an up-to-24-hour processing window with no guaranteed latency SLA,

and no multi-turn tool calling inside one batch request. Those are dated exam

reference facts, not a promise that pricing or service limits will remain

unchanged. Confirm current pricing, limits, retention, and feature compatibility

in the Message Batches documentation

before deployment.

Batch fits:

large offline extraction
evaluation datasets
nightly classification
backfills and reprocessing
independent review after generation

Real-time fits:

interactive user response
tasks with a strict short latency bound
adaptive tool use during the same request
workflows requiring immediate approval or feedback

Do not use batch when the next step depends on an external action the model must

observe mid-request. Precompute inputs or split the workflow into jobs.

Make Batch Jobs Reconciliable

Give every item a stable custom_id. Persist source version, schema version,

prompt version, and expected output location. Results may return out of order.

Handle:

success
validation failure
provider failure
expiration
duplicate submission
partial job completion
retry after source change

Never join results to inputs by array position.

Evaluate the Error You Care About

For extraction:

field precision and recall
exact or normalized match where appropriate
evidence-support rate
false-positive rate for high-risk fields
null calibration
category confusion matrix
reviewer disagreement
cost and latency per accepted record

Averages can hide a dangerous false-positive class. Stratify by document type,

language, length, and risk.

Build It

Interactive Lab

Use the confidence and review simulator to move records through syntax, schema,

semantic, and provenance gates. Adjust false-positive cost and reviewer

coverage to see why valid JSON and model confidence are insufficient release

criteria.

Practice Lab

Change one supported date to an invented value, run the four validation layers,

and route the failed record to adjudication rather than another blind retry.

Shipped Artifact

The filled outputs/extraction-review-report.md

contains a batch job with stable custom_id values, nullable unknowns, shuffled results,

review findings, and an adjudication state.

Verify It

Run its deterministic verifier:

bashcd certifications/claude/lessons/20-reliable-extraction-batch-and-reviewers
python3 code/main.py
python3 -m unittest discover -s code/tests -v

The quiz checks repair, batch, and reviewer decisions.

Capstone Connection

Carry the verified report into the Architect Foundations extraction scenario as

evidence for all four validation layers.

Create an extraction pipeline for support-policy changes.

Output Contract

Extract policy ID, effective date, affected region, action type, threshold,

evidence span, source version, and review state. Every uncertain field is

nullable or has an explicit other state.

Dataset

Build at least 40 examples:

15 clear changes
10 background statements with no change
5 negations or exceptions
5 missing dates or thresholds
5 conflicting versions

Passes

generator with strict schema
deterministic syntax and schema validation
semantic relationship validation
independent evidence reviewer
human adjudication for disagreements

Experiment

Compare zero-shot criteria, few-shot boundary examples, and generator plus

reviewer. Report false positives, evidence support, cost, and latency.

Batch Design

Submit records with stable IDs. Randomize result order in a test. Inject partial

failure and prove reconciliation keeps completed records and retries only safe

items.

Use It

In production, store the raw source separately from normalized extraction. Keep

the source version and evidence offsets. When criteria or schema change, create a

new output version rather than overwriting historical decisions.

If a human corrects a record, store a reason code. Use disagreements to improve

criteria and the evaluation set before modifying the prompt.

For high-risk extraction, review can be stratified: every high-impact field,

low-evidence record, or new document type plus a random sample of ordinary cases.

Exam Decision Patterns

When JSON is valid but content is wrong, add semantic and evidence validation.

When consistency is weak at a judgment boundary, use explicit criteria and

few-shot examples.

Prefer answers that:

use null or other rather than invention
force a typed output without triggering a real action
feed specific validation errors back with a retry limit
separate generator and reviewer
use batch for asynchronous, tool-independent workloads
reconcile results with stable IDs

Common Traps

Schema Equals Truth

Types cannot prove that a value appears in or follows from the source.

Required Non-Nullable Fields

The model invents a plausible value because the contract has no representation

for absence.

Infinite Repair

The same ambiguous source produces repeated guesses. Escalate after a bounded

attempt.

Reviewer Rewrites Silently

The system loses which claim failed and why. Return structured findings before

any controlled correction.

Exercises

Add a semantic rule linking threshold and currency.
Design negative examples that reduce false obligations.
Calibrate a reviewer against human labels and report disagreement.
Build stable-ID reconciliation for shuffled batch results.
Compare cost per accepted record for one-pass and reviewer pipelines.

Key Terms

Term	What people say	What it actually means
Structured output	Correct data	Data that matches a machine-readable shape
Semantic validation	Schema validation	Checks that values and relationships make sense for the domain
Provenance validation	Valid citation	Proof that source evidence supports the exact extracted claim
Nullable	Optional field	An explicit supported state for unknown or absent value
Batch	Faster API	Asynchronous processing optimized for offline volume and different cost or latency constraints
Adjudication	Retry	A qualified decision that resolves evaluator or label disagreement