End-to-End Eval Runner
Five lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter, scores with lessons 71 and 72, attaches the calibration report from lesson 73, and emits the leaderboard from lesson 74. Demo self-terminates. Define a ModelAdapter interface that any model (mock, local, API) can satisfy with a small method surface. Run the eval over a fixture JSONL file with parallel task execution across a worker pool. Compose the metric layer (exactmatch, F1, BLEU-4, ROUGE-L, codeexec) with the calibration layer in one pass. Emit per-model EvalRun records and feed them straight into the leaderboard aggregator. Output both a JSON report and a markdown table; self-terminate with exit zero on a clean run, non-zero on validation or runtime failure. The runner is the integration point. Each lesson 70 through 74 owns one module that the runner composes. The runner does not duplicate any logic from those modules: it imports them. The adapter is the seam between the runner and any model. The interface is intentionally small. Generation is a dataclass with: text: the model's free-form output. confidence: a float in [0, 1] representing the model's self-reported probability for the answer. tokennll: optional sum of negative log-likelihoods over the generated tokens. tokencount: optional number of generated tokens. Mock adapters in…
End-to-End Eval Runner: Five lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.