BERT, GPT and T5 Compared

Encoder-only, decoder-only and encoder-decoder side by side. Objectives, masks, blocks, decoding and when to pick each.

Three families side by side#

Factor	BERT	GPT	T5 / BART
Stack	Encoder only	Decoder only	Encoder and decoder
Attention	Bidirectional, no mask	Causal triangle mask	Bidirectional encoder, causal decoder
Cross-attention	None	None	Q from decoder, K/V from encoder
Objective	MLM on 15% of tokens	Next token, shift by one	Span corruption or denoising
Predicts	$inline$	$inline$	Target spans or the clean sequence
Generates text	No	Yes, token by token	Yes, from the encoder output
Best for	Classification, retrieval, NER	Chat, code, reasoning	Translation, speech to text
Examples	BERT, DeBERTa, ModernBERT	GPT, Llama, Qwen	T5, BART, Whisper

Task	Code
BERT 80/10/10 masking	`create_mlm_batch(tokens, vocab_size)`
Mask whole words	`whole_word_mlm(tokens, spans, vocab_size)`
GPT causal mask	`causal_mask(n)`
Next-token loss	`cross_entropy_shifted(logits, target_ids)`
T5 span corruption	`corrupt_spans(tokens, mask_rate=0.15)`
Rebuild the original	`round_trip(source, target)`
One encoder block	`encoder_block(x, p)`
One decoder block	`decoder_block(x, enc_out, p)`

tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")

model = AutoModel.from_pretrained("answerdotai/ModernBERT-base")

out = model(**inputs).last_hidden_state(1, N, 768), embeddings only

model = AutoModelForCausalLM.from_pretrained("<model>")

out = model.generate(**inputs, max_new_tokens=64,

temperature=0.7, top_p=0.9, do_sample=True)sample a reply

model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base")

inputs = tok("translate English to French: <text>", return_tensors="pt")task name in the input

out = model.generate(**inputs, max_new_tokens=32)

print(tok.decode(out[0], skip_special_tokens=True))

MLM: select 15% of tokens. 80% become [MASK], 10% random, 10% unchanged.
Labels: the original token at selected positions. -100 at all other positions.
NSP: predict if sentence B follows A. RoBERTa showed it hurt. Modern encoders skip it.
Causal LM: input [t1, t2, t3], targets [t2, t3, t4]. Sum $inline$ over positions.
Causal mask: $inline$ if $inline$ , else $inline$ . Add it before softmax.
Span corruption: spans of mean length 3, 15% total, each replaced by <extra_id_0>, <extra_id_1>.
T5 target: only the corrupted spans, each after its sentinel.
BART: five noise functions. The decoder rebuilds the full original sequence.

Command	What it does
`sample_greedy(probs)`	Argmax each step. Code completion, deterministic tasks
`sample_temperature(logits, t, rng)`	Divide logits by T. Higher T gives more diversity
`sample_top_k(logits, k, rng)`	Sample from the top k tokens only
`sample_top_p(logits, p, rng)`	Smallest set with cumulative prob at least p
`sample_min_p(logits, min_p, rng)`	Keep tokens with `p >= min_p * max_p`
Beam search	Width 4 to 5 for translation and summarization
Speculative decoding	Draft model proposes N tokens. Big model verifies

Setting	Default	Note
`mask_prob`	0.15	BERT selection rate
`IGNORE_INDEX`	-100	Label at unselected positions
`mask_rate`	0.15	T5 share of tokens in spans
`mean_span`	3.0	T5 mean span length
`mask_token`	`<mask>`	BART mask and infill token
`layer_norm` eps	1e-5	Subtracts the mean
`rms_norm` eps	1e-6	No mean subtraction
`temperature`	1.0	`softmax`, top-k, top-p, min-p

Use BERT for classification, embeddings, NER, NLI and RAG rerankers. It runs 5 to 10× faster per token than decoders.
Do not use BERT for generation. The encoder cannot produce tokens autoregressively.
Use GPT for generation, chat, code and reasoning. Decoder-only scales cleanest.
Use min-p + temperature 0.7 as a default for open-weights models.
Use T5 or BART when the input is a clear source sequence: translation, speech, structured tasks.
Use beam search with an encoder-decoder. The output distribution is narrower than chat.
Summarization and extraction work with either encoder-decoder or decoder-only.

Symptom	Cause	Fix
Model fails without `[MASK]` at fine-tune	Trained on 100% `[MASK]`	Use the 80/10/10 rule
Model cheats by looking at the answer	No causal mask	Add the `-inf` triangle before softmax
Future weights not exactly zero	Softmax never outputs an exact zero	Put `-inf` into future scores
Gradients vanish in a deep stack	No residual connections	Use `x + sublayer(x)`
Activations explode after 12 layers	Post-norm	Switch to pre-norm
Corrupted span does not rebuild	Off-by-one in span bookkeeping	Run the round-trip check
Slow decode on long inputs	Encoder runs again each step	Cache the encoder output

Why can BERT use the right context?: The encoder has no causal mask.
Why is GPT training parallel?: One pass gives N next-token losses. Inference stays serial.
How many params in one block?: MHA is 4 · d². SwiGLU FFN is about 3rd².
Why is T5 pretraining cheaper than BART?: T5 decodes only the spans. BART decodes the full sequence.
Are embedding models BERT?: Yes. Same encoder, contrastive loss.