Language Model Evaluation Harness
A model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one short, swappable shape. Define a task as a JSONL file with prompt, targets, metric, and optional extras per example. Implement five metrics: exact match, rouge-l F1, executable check, multiple choice, and substring contains. Build a runner that batches examples per task and dispatches to a swappable model adapter. Emit a leaderboard JSON with per-task scores, latency, and an overall average that is reproducible. A new language model lands every week. The marketing claim is that it does well. The honest question is: well at what? The honest answer is the leaderboard you wrote yourself, because the vendor's leaderboard is the one they tuned to. Without a harness in your repo you compare two models by vibes. With a harness you compare them by score on a fixed task set with a fixed metric, on a JSON output you can diff. The harness is the contract between yesterday's run and today's run. Without it, regressions ship. The trap is over-fitting the harness to a single model. The fix is the same trap in reverse: the harness is small enough to read in fifteen minutes, the tasks are…
Language Model Evaluation Harness: A model that does well on a task you cannot define is a model that does well by accident. The harness is the task…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.