Build a Transformer from Scratch — The Capstone
Thirteen lessons. One model. No shortcuts. You've read every paper. You've implemented attention, multi-head splits, positional encodings, encoder and decoder blocks, BERT and GPT losses, MoE, KV cache. Now make them work together on a real task. The capstone: train a small decoder-only transformer end-to-end on a character-level language modeling task. It reads Shakespeare. It generates new Shakespeare. It is small enough to train on a laptop in under 10 minutes. It is correct enough that swapping in a bigger dataset and longer training gets you a real LM. This is the "nanoGPT" of the course. It is not original — Karpathy's 2023 nanoGPT tutorial is the reference implementation every student writes at least once. We lift the shape and retool it around what we've covered. Transformer-from-scratch block diagram The architecture, annotated: GPTConfig — one place to configure all hyperparameters. MultiHeadAttention — causal, batched, with optional Flash-style pathway (PyTorch's scaleddotproductattention). SwiGLUFFN — modern FFN. Block — pre-norm, residual-wrapped attention + FFN. GPT — embeddings, stacked blocks, LM head, generate(). Training loop with AdamW, cosine LR, gradient clipping. Char-level tokenizer on Shakespeare text. RoPE — implemented conceptually in Lesson 04. Here we use learned positional embeddings for simplicity. The exercises ask you to swap in RoPE. KV cache during generation — each generation step recomputes attention over the full prefix. Slower…
Build a Transformer from Scratch — The Capstone: Thirteen lessons. One model. No shortcuts. You've read every paper. You've implemented attention, multi-head…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.