What is Regression Test?

A repeatable check that protects behavior known to work, especially after code, prompt, model, retrieval, or tool changes.

Why does Regression Test matter?

AI system changes can improve average quality while silently reintroducing a previously fixed failure.

Turn a corrected prompt-injection incident into a permanent eval case that must pass before the next deployment.

A regression test guards a specific expected behavior. A broad benchmark estimates performance across a wider task distribution.

Start with

Eval-Driven Agent Development
Anthropic's guidance: "start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed." Evaluation is not the last step.
Phase 14: Agent Engineering

Eval SetA versioned collection of inputs, expected properties, scoring rules, and metadata used to measure an AI system against a defined…
Verification GateA control point that blocks progress until defined evidence satisfies a correctness or quality criterion.
PatchA reviewable representation of changes to one or more files, usually expressed as additions and deletions against a known base revision.
Evaluation (Eval)A defined process for measuring model or system behavior on representative tasks using explicit success criteria, data, scorers, and…
Flaky TestA test that can pass and fail across equivalent runs without a relevant change to the code or intended test environment.
Pass@kAcross a task set, the fraction of tasks for which at least one of k sampled candidates passes a defined correctness test.
PostmortemA durable incident record that explains impact, detection, response, contributing conditions, recovery, and owned follow-up actions…
Prompt SensitivityVariation in model output or measured performance caused by changes to prompt wording, order, formatting, or examples that preserve the…
RollbackRestoring a previously known deployment or configuration when the current release violates operational, quality, or safety criteria.
Test OracleThe mechanism, specification, reference, invariant, or human judgment used to decide whether observed program behavior is correct.