What is Distribution Shift?

A difference between the data distribution used to build or evaluate a system and the distribution it encounters after deployment.

Why does Distribution Shift matter?

A model can pass held-out tests yet fail when users, tasks, language, tools, or operating conditions change.

Define expected deployment slices, monitor performance and input characteristics by slice, and add new failures to a versioned eval set.

Distribution shift is not always model drift. The model may be unchanged while its environment or user population changes.

No lesson links to this term yet. Search the course catalog for it.

Dataset SplitA documented partition of examples into separate subsets for fitting, development decisions, and final evaluation.
Eval SetA versioned collection of inputs, expected properties, scoring rules, and metadata used to measure an AI system against a defined…
OverfittingA generalization gap in which performance on training data is substantially better than performance on representative unseen data.
Model CardA structured report describing a model's intended uses, evaluation conditions, performance characteristics, limitations, and relevant…