Whisper — Architecture & Fine-Tuning
Whisper is a 30-second-window transformer encoder-decoder, trained on 680k hours of multilingual weakly-supervised audio-text pairs. One architecture, multiple tasks, robust across 99 languages. The 2026 reference ASR. Whisper, released by OpenAI in September 2022, was the first ASR model to ship as a commodity: paste audio, get text, 99 languages, robust to noise, runs on a laptop. By 2024 OpenAI had shipped Large-v3 and Turbo variants; by 2026, Whisper is the default baseline for everything from podcast transcription to voice assistants to YouTube subtitles. But Whisper is not a pipeline you can treat as a black box forever. Domain shift kills it — technical jargon, speaker accents, proper nouns, short clips, silence. You need to know: What it actually is inside. How to give it chunked, streaming, or long-form audio correctly. When to fine-tune and how. Whisper encoder-decoder, tasks, chunked inference, fine-tune Architecture. Standard transformer encoder-decoder. Input: 30-second log-mel spectrogram, 80 mels, 10 ms hop → 3000 frames. Clips shorter are zero-padded, clips longer are chunked. Encoder: conv-downsample (stride 2) + N transformer blocks. For Large-v3: 32 layers, 1280-dim, 20 heads. Decoder: N transformer blocks with causal self-attn + cross-attn to encoder output. Same size as encoder. Output: BPE tokens over a 51,865-token vocab. Large-v3 has 1.55B params. Turbo uses a 4-layer decoder (from 32), cutting latency 8× with a…
Whisper — Architecture & Fine-Tuning: Whisper is a 30-second-window transformer encoder-decoder, trained on 680k hours of multilingual weakly-supervised…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.