GPT — Causal Language Modeling
BERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI. A language model answers one question: given the first t-1 tokens, what is the probability distribution over token t? Train on that signal — next-token prediction — and you get a model that can generate arbitrary text one token at a time. To train it end-to-end on a whole sequence in parallel, you need each position's prediction to depend only on earlier positions. Otherwise the model trivially cheats by looking at the answer. The causal mask does this. It is a single upper-triangular matrix of -inf values added to attention scores before softmax. After softmax, those positions become 0. Each position can attend only to itself and earlier positions. And because you apply it once to the whole sequence, you get N parallel next-token predictions in one forward pass. GPT-1 (2018), GPT-2 (2019), GPT-3 (2020), GPT-4 (2023), GPT-5 (2025), Claude, Llama, Qwen, Mistral, DeepSeek, Kimi — they are all decoder-only causal transformers with the same core loop. What separates them is data quality, scale and architectural refinements, and post-training (SFT, RLHF, DPO, and their successors). Causal mask creates a triangular attention matrix Given a sequence of length N, build an N × N matrix: Add M to…
GPT — Causal Language Modeling: BERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.