The Full Transformer — Encoder + Decoder
Attention is the star. Everything else — residuals, normalization, feed-forward, cross-attention — is the scaffolding that lets you stack it deep. A single attention layer is a feature extractor, not a model. One matmul per layer is not enough capacity for language. You need depth — and depth breaks without the right plumbing. The 2017 Vaswani paper packaged six design decisions that turned one attention layer into a stackable block. Every transformer since — encoder-only (BERT), decoder-only (GPT), encoder-decoder (T5) — inherits the same skeleton. In 2026 the blocks have been refined (RMSNorm, SwiGLU, pre-norm, RoPE) but the skeleton is identical. This lesson is the skeleton. Next lessons specialize it — 06 for encoders, 07 for decoders, 08 for encoder-decoder. Encoder and decoder block internals, wired Embedding + positional signal. Tokens → vectors. Position injected via RoPE (modern) or sinusoidal (classic). Self-attention. Every position attends to every other. Masked in decoders. Feed-forward network (FFN). Position-wise two-layer MLP: W2 · activation(W1 · x). Expansion ratio 4× by default. Residual connection. x + sublayer(x). Without this, gradients vanish past 6 layers. Layer normalization. LayerNorm or RMSNorm (modern). Stabilizes the residual stream. Cross-attention (decoder only). Queries come from the decoder, keys and values from the encoder output. Watch a vector flow through one block: attention mixes across positions, the residual carries it forward,…
The Full Transformer — Encoder + Decoder: Attention is the star. Everything else — residuals, normalization, feed-forward, cross-attention — is the…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.