Self-Attention from Scratch
Attention is a lookup table where every word asks "who matters to me?" - and learns the answer. Implement scaled dot-product self-attention from scratch using only NumPy, including query/key/value projections and the softmax-weighted sum. Build a multi-head attention layer that splits heads, computes parallel attention, and concatenates results. Trace how the attention matrix captures token relationships and explain why scaling by sqrt(dk) prevents softmax saturation. Apply causal masking to convert bidirectional attention into autoregressive (decoder-style) attention. RNNs process sequences one token at a time. By the time you reach token 50, the information from token 1 has been squeezed through 50 compression steps. Long-range dependencies get crushed into a fixed-size hidden state - a bottleneck that no amount of LSTM gating fully solves. The 2014 Bahdanau attention paper showed the fix: let the decoder look back at every encoder position and decide which ones matter for the current step. But it was still bolted onto an RNN. The 2017 "Attention Is All You Need" paper asked a sharper question: what if attention is the only mechanism? No recurrence. No convolution. Just attention. Self-attention lets every position in a sequence attend to every other position in a single parallel step. That is what makes transformers fast, scalable, and dominant. Think of attention as a soft database lookup: Every token generates three…
Self-Attention from Scratch: Attention is a lookup table where every word asks "who matters to me?" - and learns the answer. Implement scaled dot-product…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.