Evaluation: Benchmarks, Evals, LM Harness
Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Every frontier lab games benchmarks. MMLU scores go up while models still can't reliably count the number of R's in "strawberry." The only eval that matters is YOUR eval -- on YOUR task, with YOUR data. Build a custom evaluation harness that runs multiple-choice and open-ended benchmarks against a language model. Explain why standard benchmarks (MMLU, HumanEval) saturate and fail to differentiate frontier models. Implement task-specific evals with proper metrics: exact match, F1, BLEU, and LLM-as-judge scoring. Design a custom evaluation suite targeting your specific use case rather than relying solely on public leaderboards. MMLU was published in 2020 with 15,908 questions across 57 subjects. Within three years, frontier models saturated it. GPT-4 scored 86.4%. Claude 3 Opus scored 86.8%. Llama 3 405B scored 88.6%. The leaderboard compressed into a 3-point range where differences are statistical noise, not real capability gaps. Meanwhile, those same models fail at tasks that a 10-year-old handles without thinking. Claude 3.5 Sonnet, scoring 88.7% on MMLU, initially could not count the letters in "strawberry" -- a task that requires zero world knowledge and zero reasoning, just character-level iteration. HumanEval tests code generation with 164 problems. Models score 90%+ on it while still producing code that crashes on edge cases any…
Evaluation: Benchmarks, Evals, LM Harness: Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Every frontier lab games…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.