AI-native development · Glossary term
What is Flaky Test?
A test that can pass and fail across equivalent runs without a relevant change to the code or intended test environment.
Why does Flaky Test matter?
Flakiness weakens verification gates and can train people or agents to ignore real failures or retry until they obtain a false pass.
Flaky Test in practice
Preserve the failing seed and environment, quarantine only with an owner and deadline, then fix uncontrolled time, concurrency, network, order, or shared-state dependencies.
What is the common confusion about Flaky Test?
A test that consistently exposes an intermittent product bug is valuable evidence, not necessarily a flaky test.
Learn Flaky Test in the course
No lesson links to this term yet. Search the course catalog for it.
Related terms
- Regression TestA repeatable check that protects behavior known to work, especially after code, prompt, model, retrieval, or tool changes.
- Test OracleThe mechanism, specification, reference, invariant, or human judgment used to decide whether observed program behavior is correct.
- Retry with BackoffRepeating a failed transient operation after progressively longer delays, usually with randomized jitter and a strict retry limit.
- Verification GateA control point that blocks progress until defined evidence satisfies a correctness or quality criterion.
Sources
More terms in AI-native development
This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.