T5, BART — Encoder-Decoder Models
Encoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe. Decoder-only GPT and encoder-only BERT each strip down the 2017 architecture for a different goal. But many tasks are naturally input-output: Translation: English → French. Summarization: 5,000-token article → 200-token summary. Speech recognition: audio tokens → text tokens. Structured extraction: prose → JSON. For these, encoder-decoder makes the cleanest fit. The encoder produces a dense representation of the source. The decoder generates the output, cross-attending to that representation at every step. Training is shift-by-one on the output side. Same loss as GPT, just conditioned on the encoder output. Two papers defined the modern playbook: T5 (Raffel et al. 2019). "Text-to-Text Transfer Transformer." Every NLP task reframed as text-in, text-out. Single architecture, single vocabulary, single loss. Pretrained on masked span prediction (corrupt spans in the input, decode them in the output). BART (Lewis et al. 2019). "Bidirectional and Auto-Regressive Transformer." Denoising autoencoder: corrupt input in multiple ways (shuffle, mask, delete, rotate), ask the decoder to reconstruct the original. In 2026 the encoder-decoder format lives on where input structure matters: Whisper (speech → text). Google's translation stack. Some code-completion / repair models that have distinct context-and-edit structures. Flan-T5 and variants for structured reasoning tasks. Decoder-only won the spotlight, but…
T5, BART — Encoder-Decoder Models: Encoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks:…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.