Capstone Lesson 41: Full Evaluation Pipeline
Training is the part you can monitor with loss curves. Evaluation is the part you have to design. This lesson builds a unified eval pipeline that takes any trained language model, runs four heterogeneous evals on it, aggregates the results into a per-task report, and ships a local mock LLM-as-judge so the loop runs without a network. The four evals cover the dimensions every shipping model needs: language modelling (perplexity), short-form correctness (exact-match), open-form similarity (token F1), and qualitative scoring (judge). Compute held-out perplexity with masked-token accounting on a tiny transformer. Run an exact-match eval on short-form factual prompts. Compute token-level F1 between predicted and reference strings with normalisation. Build a local mock LLM-as-judge that scores model outputs on a 1-5 scale. Aggregate the four evals into a single weighted report with per-task breakdown. A single metric never describes a language model. Perplexity says how well the model fits the language distribution but says nothing about whether it answers questions. Exact-match says whether the model produces the gold string but punishes correct paraphrases. Token F1 forgives paraphrase but is fooled by lexical overlap with wrong content. LLM-as-judge captures qualitative dimensions but is expensive and stochastic. The pipeline you actually want has all four. Each eval covers a dimension the others miss. Each runs on a different subset of held-out data…
Capstone Lesson 41: Full Evaluation Pipeline: Training is the part you can monitor with loss curves. Evaluation is the part you have to design. This lesson…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.