What is Test Oracle?

Why does Test Oracle matter?

Generating test inputs is not enough; automated verification requires an independent basis for classifying each result.

Test Oracle in practice

Prefer executable invariants, reference implementations, schemas, and deterministic expected outputs, then document where human judgment remains necessary.

What is the common confusion about Test Oracle?

The model that wrote the code should not be treated as an independent oracle merely because you ask it whether its own output is correct.

Learn Test Oracle in the course

No lesson links to this term yet. Search the course catalog for it.

Related terms

• Regression TestA repeatable check that protects behavior known to work, especially after code, prompt, model, retrieval, or tool changes.
• Verification GateA control point that blocks progress until defined evidence satisfies a correctness or quality criterion.
• Eval SetA versioned collection of inputs, expected properties, scoring rules, and metadata used to measure an AI system against a defined…
• Human-in-the-Loop (HITL)A workflow design in which a person supplies judgment, correction, approval, or escalation at defined points in an AI-driven process.
• Flaky TestA test that can pass and fail across equivalent runs without a relevant change to the code or intended test environment.
• Pass@kAcross a task set, the fraction of tasks for which at least one of k sampled candidates passes a defined correctness test.

Sources

• The Oracle Problem in Software Testing

More terms in AI-native development

• Backpressure
• Circuit Breaker
• Coding Agent
• Cost per Successful Task
• Flaky Test
• Handoff
• Idempotency
• Model Router
• Observability
• Patch
• Progressive Disclosure
• Rate Limit
• Regression Test
• Repository Instructions
• Repository Map
• Reproducible Build
• Retry with Backoff
• Reviewer Agent
• Scope Contract
• Semantic Cache
• Trace
• Worktree

Open the AI-native development list in the glossary