Perplexity and Calibration
If your model says 90 percent confident on a thousand answers and gets six hundred right, it is not well calibrated. Calibration is half of trustworthy eval. The other half is perplexity, which tells you whether the model thinks the held-out text is plausible at all. Compute token-level perplexity on a held-out corpus from token negative log-probabilities supplied by the model adapter. Compute the expected calibration error (ECE) of a classifier or multiple-choice eval from binned predicted probabilities. Compute the Brier score (mean squared error against the indicator of correctness) and explain when it does what ECE does not. Build the reliability diagram data needed to plot a confidence-versus-accuracy curve. Wire all three into the eval harness so the runner can attach perplexity, ece, and brier numbers to a model report. Perplexity is the exponentiated average negative log-likelihood per token. Lower is better. A perplexity of one means the model assigns probability one to every actual token. A perplexity of the vocabulary size means the model is uniform and learnt nothing. Real numbers fall in between: a strong 2026 base model on WikiText-103 sits around eight to twelve. A bad one on the same text sits at fifty plus. The harness does not compute log-probabilities itself. Those come from the model adapter. The harness aggregates: it takes a list of…
Perplexity and Calibration: If your model says 90 percent confident on a thousand answers and gets six hundred right, it is not well calibrated. Calibration…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.