Loading Pretrained Weights
Training a 124 million parameter model from scratch is a budget decision; loading a published checkpoint is a Tuesday. This lesson loads pretrained GPT-2 style weights from a safetensors file into the exact architecture from lesson 35, walks the parameter name mapping piece by piece, and sanity generates a continuation to prove the load worked. No network, no third party loaders, no opaque magic. Read a safetensors file with the safetensors Python library and inspect the tensor names and shapes. Map each pretrained parameter name onto a parameter inside the lesson 35 GPT model. Handle the two name conventions that differ between published GPT-2 weights and the model in this track: wte/wpe/h.N.attn.cattn/cproj and mlp.cfc/cproj versus the locally named tokembed/posembed/blocks.N.attn.qkv/outproj and mlp.fc1/fc2. Detect and refuse a shape mismatch with a clear error before any weight assignment happens. Generate a short continuation with the loaded weights and confirm the tokens come from the loaded distribution, not the randomly initialized one. Published weights are not packaged for your architecture. They carry the names the original implementation used. The pretrained file has transformer.h.0.attn.cattn.weight of shape (2304, 768); your model expects blocks.0.attn.qkv.weight of shape (2304, 768) (which is the same matrix in a different layout convention) or your model uses nn.Linear which stores the matrix transposed. The same parameter shows up with three subtly different…
Loading Pretrained Weights: Training a 124 million parameter model from scratch is a budget decision; loading a published checkpoint is a Tuesday. This lesson…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.