Cheat sheets

Evaluating LLMs

Specify tasks, score them, judge with a rubric, calibrate, rank with intervals and test long context.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

End-to-end eval runner#

rendertextcode_execscorescoreconfidencereporttasks.jsonlvalidate_fileModelAdaptergenerate(prompt,task)post_processone rule per taskscoremetric_name dispatchCalibrationReportece, brieraggregateJSON + markdownEvalRunone per model, taskrun_candidatesubprocess, 3 srendertextcode_execscorescoreconfidencereporttasks.jsonlvalidate_fileModelAdaptergenerate(prompt, task)post_processone rule per taskscoremetric_name dispatchCalibrationReportece, brieraggregateJSON + markdownEvalRunone per model, taskrun_candidatesubprocess, 3 s

From Lesson 19.75, 19.75 code/main.py, 19.72 code/main.py

Score one task#

1Renderfew-shot examples, then the prompt, with a blankline between.2Adaptergenerate(prompt, task) returns text andconfidence. The runner times it.3post_processapplies the single rule of the task.4Dispatchscore for text metrics, run_candidate forcode_exec.5EvalRunone record with the score and the metric name.6Calibration bufferappends the (confidence, correct) pair.7Correctscore of 1 for exact_match, accuracy, code_exec.At least 0.5 for graded metrics.

From Lesson 19.75, 19.75 code/main.py, Lesson 19.70

Pick and run a metric#

TaskCode
Score arithmetic"exact_match" with "strip_whitespace"
Score multiple choice"exact_match" with "extract_letter"
Score generated code"code_exec" with "extract_code_block"
Score a summary"rouge_l" or "bleu_4"
Interval on a pass ratewilson_confidence_interval(successes, total, z=1.96)
Rank by pairwise winsELOTracker(k=32, initial_rating=1500)
Plant a needlebuild_haystack(filler_text, needle, depth_ratio, total_tokens)
Run a public benchmarklm_eval --model hf --model_args pretrained=<model> --tasks mmlu

From 19.70 code/main.py, 19.71 code/main.py, 11.10 code/eval_framework.py, Lesson 10.10, Lesson 05.28

Harness API#

tasks, errors = validate_file("tasks.jsonl")do not run if errors
s = score("f1", prediction, targets)best F1 over the targets
s = score_code_exec(prediction, assertions)passed / total, 3 s limit
p = pass_at_k(n=10, c=3, k=5)unbiased estimator
report = CalibrationReport.from_predictions(confidences, correct)
report.ece, report.brier, report.populated_binsbins=10 by default
ppl = perplexity(neg_log_probs, token_counts)exp(total nll / tokens)
rows = aggregate(runs, b=500, alpha=0.05)mean, bootstrap CI, win rate
print(render_markdown(rows))table for a PR comment

From 19.70 code/main.py, Lesson 19.70, 19.71 code/main.py, 19.72 code/main.py, 19.73 code/main.py, 19.74 code/main.py

Post-process rules#

CommandWhat it does
noneReturns the string unchanged
strip_whitespaceStrips leading and trailing whitespace
lowerLowercases the string
extract_letterFirst character in [A-E]. For MCQ
extract_code_blockBody of the first fenced block. For code
extract_first_lineFirst non-empty line. For summary classes

From Lesson 19.70, 19.70 code/main.py

Sample size and confidence#

Test cases at 90%95% CI width
5019 points. Cannot detect a 5% regression
10012 points. Barely
2009 points. Yes
5005 points. Confidently
10003 points. Precisely

From Lesson 11.10

Tips#

  • Anchor each rubric score to observable behavior. Judge variance drops 30-40%.
  • Freeze the judge model and its version.
  • Pin task files by hash in the leaderboard.
  • Diff predictions, not only scores. Use --include-per-example.
  • Keep eval cases apart from prompt and training examples.
  • Test needle depths 0, 0.25, 0.5, 0.75 and 1.0.

From Lesson 11.10, Lesson 05.27, Lesson 19.49, Lesson 05.28

Key formulas#

Perplexity
\mathrm{PPL} = \exp\left(-\frac{1}{N} \sum_i \log P(\text{token}_i \mid \text{context})\right)
Token F1
\frac{2 P R}{P + R}. P and R are token overlap over prediction and target.
BLEU-4
brevity penalty times the geometric mean of p_1 to p_4.
Brevity penalty
1 if c \ge r, else \exp(1 - r / c).
ROUGE-L
LCS over reference length and over candidate length, then F1.
pass@k
1 - \binom{n - c}{k} / \binom{n}{k} for c passes in n samples.
ECE
confidence minus accuracy per bin, weighted by bin size, summed.
Brier
\mathrm{mean}((p_i - y_i)^2). Penalizes spread that ECE misses.

From Lesson 10.10, Lesson 19.71, Lesson 19.72, Lesson 19.73

Defaults in the code#

SettingDefaultNote
DEFAULT_TIMEOUT_S3.0Per task. MAX_TIMEOUT_S is 30.0
OUTPUT_CAP_BYTES256 * 1024Then the result is "output overflow"
FEW_SHOT_MAX8Few-shot examples per task
ECE bins1015 is the comparison convention
Bootstrap b500Low-level helpers use 1000
alpha0.0595% percentile interval
max_workers8Thread pool in run_eval
ELO k32Start rating 1500

From 19.72 code/main.py, 19.70 code/main.py, 19.73 code/main.py, Lesson 19.73, 19.74 code/main.py, 19.75 code/main.py, Lesson 10.10

Troubleshooting#

SymptomCauseFix
Perplexity below 1Adapter returns log p, not -log pNegate. The function rejects it
Perplexity too goodPadding counted in the denominatorCount only non-pad tokens
Judge scores go missingBad JSON gives NaN, mean drops itCatch it as an explicit failure
Judge rates its own model highSelf-evaluation adds 10-20%Judge with a different model family
Pairwise judge picks the firstPosition biasRandomize order and run both
Mean 0.85, users complainMean hides 5% catastrophic failuresInspect the bottom quantile
Passes NIAH, fails real docsNIAH tests retrieval onlyAdd RULER or a multi-hop test
One task dominates the meanScore in [0, 100]Return fractions in [0, 1]

From Lesson 19.73, Lesson 19.41, Lesson 05.27, Lesson 05.28, Lesson 19.74

Abbreviations#

EM
Exact Match. 1 or 0 per task after normalization.
ECE
Expected Calibration Error. Gap between confidence and accuracy.
MMLU
Massive Multitask Language Understanding. 15,908 questions, 57 subjects.
HumanEval
164 Python function-completion problems.
SWE-bench
2,294 GitHub issues from 12 Python repos.
NIAH
Needle in a Haystack. Plant a fact in filler, ask for it.
MRCR
Multi-Round Coreference Resolution. 8, 24 or 100 needles.
RAGAS
Retrieval-Augmented Generation ASsessment. Four RAG metrics.

From Lesson 05.27, Lesson 19.73, Lesson 10.10, Lesson 05.28

Questions#

How many test cases?
At least 200 for a deploy decision. 500 or more for close systems.
Why report pass@1 too?
pass@k hides a model that is right once in twenty tries.
When is a difference real?
When the bootstrap interval of the paired difference excludes zero.
When can I trust a judge?
Spearman rho against human labels of 0.7 or more.
How long is the real context?
Reasoning-effective is often 25-50% of the advertised window.

From Lesson 11.10, Lesson 19.27, Lesson 19.74, Lesson 05.27, Lesson 05.28