Leaderboard Aggregation
Per-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction leaderboard is the part everyone skips. This lesson does not skip it. Aggregate per-task scores across multiple models and multiple tasks into a tidy per-model row. Normalise heterogeneous scores so that pass rates and BLEU values do not over-influence the aggregate. Rank models by mean and by win-rate, and explain when each is the right summary. Compute bootstrap confidence intervals on the mean score per model and on pairwise differences. Output the leaderboard as a JSON report and as a markdown table the runner in lesson 75 can paste into a CI comment. The aggregator consumes a list of EvalRun records: The runner in lesson 75 emits one record per (model, task) pair. The aggregator does not care how the score was produced. It expects normalisation to already have happened: every score is in [0, 1]. Three tables come out: The leaderboard row contains: modelid, meanscore, meancilo, meancihi, winrate, taskscompleted, and an optional categories map for per-category mean. If one task scores in [0, 1] and another in [0, 100], the second silently dominates the mean. The aggregator validates that every input score sits in [0, 1] and refuses the run otherwise. The fix lives upstream: the metric should already return a fraction. Lessons…
Leaderboard Aggregation: Per-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.