Encoder, decoder or both for three products
RolesLLM Product EngineeringCustomer AI DeploymentAI Data SystemsInference Engineering
The question
Three products land on your desk: a search ranking model over documents, a coding assistant that writes code, and a speech-to-text system. Which transformer shape do you pick for each, and what does the pretraining loss have to do with the choice?
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
Encoder-only for ranking and embeddings, decoder-only for generation, and encoder-decoder when the input is a clear source sequence such as audio to transcribe. An encoder reads both sides of every position and trains on masked prediction, which gives strong representations and no way to generate. A decoder trains on next-token prediction behind a causal mask, so it can generate, and it scales cleanest, which is why it won language. The loss decides whether the model can produce text at all.
Show full answer
The three skeletons
| Need | Shape | Why |
|---|---|---|
| Classification, embeddings, question answering over text | Encoder-only | Bidirectional attention sees the whole input at once |
| Text generation, chat, code, reasoning | Decoder-only | Causal mask plus next-token loss; one pass of attention and FFN per layer |
| Structured input to structured output, translation, speech recognition | Encoder-decoder | The encoder owns the source; the decoder generates and reads it through cross-attention |
The one line that separates them
A decoder predicts each token's probability given the tokens before it. An encoder predicts a masked token given the unmasked ones. Same block, same attention, different mask and different target. Only the first loss produces a model that can continue a sequence.
Cross-attention
A decoder block in an encoder-decoder model has three sublayers: masked self-attention, cross-attention and the FFN. In cross-attention the queries come from the decoder and the keys and values come from the encoder output. It is the only place where information flows from the source to the output. A pure decoder drops it.
Why decoder-only won language
It handles comprehension and generation with one architecture, it scales cleanly, and in-context learning let one model follow examples without a new head per task. Encoder-decoder is still the best fit when the input has a strong source identity: a waveform, a sentence in another language, a table to summarize.
The three answers
- Ranking: an encoder, or an embedding model built on one, because you need a representation of the whole document and no generation.
- Coding assistant: a decoder, because the product is generation and the causal loss is what makes generation possible.
- Speech-to-text: an encoder-decoder, because the audio is a source sequence of its own kind and the text is a different sequence.
Follow-ups
L2WorkingWhich sublayer exists only in the encoder-decoder shape?
Show expected answer
cross-attention, with queries from the decoder and keys and values from the encoder.
L3SeniorWhy can a decoder-only model also do comprehension tasks?
Show expected answer
the same next-token model answers a question by continuing a prompt, and in-context learning lets it follow examples without a task head.
L3SeniorWhen would you still choose encoder-decoder for a text task?
Show expected answer
when the input is a clear source sequence that the output transforms, such as translation or summarization of a fixed document.
L4StaffThe ranking team wants to reuse the decoder they already serve. How?
Show expected answer
prompt it and read the logits of a label token, or attach a head and fine-tune; weigh the cost against an encoder that scores documents in one pass.
Wrong answers
Show the wrong answers that interviewers hear
“A decoder-only model cannot classify.”
It can, by prompting or by adding a head; the point is that an encoder is often cheaper for that job.
“An encoder generates text one token at a time.”
It has no causal loss and predicts masked positions; it is not trained to continue a sequence.
“Every transformer block has cross-attention.”
Only the decoder blocks of an encoder-decoder model have it.
Taught in
The full transformer lesson gives the three shapes and the picking table, and the causal language modeling lesson contrasts the two losses.