Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards
You cannot ship what you cannot measure. This lesson names the 2026 metrics for every audio task: ASR (WER, CER, RTFx), TTS (MOS, UTMOS, SECS, WER-on-ASR-round-trip), audio-language (MMAU, LongAudioBench), music (FAD, CLAP), and speaker (EER). Plus the leaderboards where you compare. Every audio task has multiple metrics, each measuring a different axis. Using the wrong metric is how you ship a model that looks great on your dashboard and terribly in production. The 2026 canonical list: Audio evaluation matrix — metrics vs tasks vs 2026 leaderboards WER (Word Error Rate). (S + D + I) / N. Lowercase, strip punctuation, normalize numbers before scoring. Use jiwer or OpenAI's whispernormalizer. < 5% = human-parity read speech. CER (Character Error Rate). Same formula, character-level. Used for tone languages (Mandarin, Cantonese) where word segmentation is ambiguous. RTFx (inverse real-time factor). Audio seconds processed per wall-clock second. Higher is better. Parakeet-TDT hits 3380×. Whisper-large-v3 is 30×. First-token latency. Wall-clock from audio input to first transcript token. Critical for streaming. Deepgram Nova-3: 150 ms. MOS (Mean Opinion Score). 1-5 human rating. Gold standard but slow. Collect 20+ listeners per sample, 100+ samples per model. UTMOS (2022-2026). Learned MOS predictor. Correlates 0.9 with human MOS on standard benchmarks. F5-TTS: UTMOS 3.95; ground truth: 4.08. SECS (Speaker Encoder Cosine Similarity). For voice cloning. ECAPA embedding cosine between…
Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards: You cannot ship what you cannot measure. This lesson names the 2026 metrics for…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.