Evaluation & safety · Glossary term
What is Benchmark Contamination?
Overlap or information leakage between evaluation examples and data used to pretrain, tune, prompt, select, or otherwise improve the evaluated system.
Why does Benchmark Contamination matter?
Contamination can make a benchmark score reflect prior exposure rather than the ability to generalize to unseen tasks.
Benchmark Contamination in practice
Track dataset provenance, search training sources for exact and near duplicates, hold back private test cases, and refresh public evals with newly authored examples.
What is the common confusion about Benchmark Contamination?
Contamination is broader than exact copying. Paraphrases, answer keys, benchmark metadata, and repeated prompt tuning can also leak evaluation information.
Learn Benchmark Contamination in the course
No lesson links to this term yet. Search the course catalog for it.
Related terms
- Data LeakageUnintended use of information during training or feature construction that would not be available at the real prediction point or belongs…
- Data DeduplicationDetecting and removing exact and near-duplicate examples within or across datasets.
- Eval SetA versioned collection of inputs, expected properties, scoring rules, and metadata used to measure an AI system against a defined…
- Exact Match (EM)A metric that counts an output as correct only when its normalized representation exactly equals an accepted reference answer.
Sources
More terms in Evaluation & safety
This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.