Result Evaluator
The runner produced numbers. The evaluator decides whether those numbers are an improvement, a regression, or noise. Build the verdict path that turns metrics into a one line conclusion. Compare a candidate run against a baseline using direction aware improvement and a fixed threshold. Run a paired t test from scratch over per seed metrics and read the resulting p value. Normalise log scaled metrics so a downstream report can blend them with linear metrics. Emit a per hypothesis verdict that the orchestrator can attach to the queue from lesson fifty. Keep every step pure so the same inputs always produce the same verdict. A single number from the runner does not say whether the change is real. The same configuration with a different seed gives a different perplexity. The change might be noise. The right comparison is paired: the same seeds with the same data, ran once with the candidate and once with the baseline. Each seed contributes a difference. The mean of those differences is the effect. The standard error of those differences is the noise floor. The lesson implements the test from scratch. There is no scipy.stats. The math is small enough to read in one screen. The two sided p value uses a regularised incomplete beta function. The lesson ships a small implementation that uses the…
Result Evaluator: The runner produced numbers. The evaluator decides whether those numbers are an improvement, a regression, or noise. Build the verdict path…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.