Reliable Extraction, Batch, and Independent Reviewers
Valid JSON proves the shape survived. It does not prove the facts did. Define extraction criteria that reduce false positives and ambiguous labels. Use schemas, examples, nullable fields, enums, and evidence spans deliberately. Separate syntax, schema, semantic, and provenance validation. Design bounded retry and independent reviewer passes. Choose real-time or batch processing from workflow requirements. A pipeline extracts contract obligations into valid JSON. Every record matches the schema. Legal reviewers still reject 18 percent. The model fills missing dates with plausible values, labels background statements as obligations, and maps unfamiliar categories to the nearest enum. A retry loop feeds the same prompt back until validation passes. Since validation checks only types, the invented values become more confidently formatted. The team solved serialization and mistook it for correctness. A schema says what fields exist. Criteria say what qualifies. For an obligation extractor, define: obligated party must be explicit or unambiguously linked. required action must be stated, not merely discussed. trigger and deadline are extracted only when supported. evidence span must contain the claim. unknown values remain null. unsupported category uses other with a note or triggers review. exceptions and negations change the result. Without these rules, annotators, model, and evaluator apply different tasks. Examples are most useful where reasonable people make different judgments. Include: a clear positive. a near miss. a…
Reliable Extraction, Batch, and Independent Reviewers: Valid JSON proves the shape survived. It does not prove the facts did.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.