Training Loop and Evaluation
A loop that does not measure is a loop that lies. This lesson builds the training loop that drives the GPT model: AdamW with weight decay split, a warmup plus cosine learning rate schedule, a calclossbatch helper, an evaluatemodel pass on held out data, a generateandprintsample qualitative probe every K steps, and a JSONL log of losses you can plot after. The same skeleton trains every decoder LLM you will ever build. Build a training loop that computes cross entropy loss with the correct input and target alignment for next token prediction. Configure AdamW with weight decay applied to weight tensors and not to LayerNorm or bias tensors. Implement a learning rate schedule with linear warmup and cosine decay, and read the resulting LR over time. Evaluate on a held out split with evaluatemodel so the eval loss is comparable across runs. Generate a qualitative sample every K steps with generateandprintsample to catch divergence before the loss curve does. Persist per step loss to JSONL so you can reload, plot, and ship the training log as a deliverable. A training script that prints the loss but does nothing else fails three ways. It cannot tell you if the loss is decreasing for the right reason (the model could overfit the training set and never learn). It cannot tell you if…
Training Loop and Evaluation: A loop that does not measure is a loop that lies. This lesson builds the training loop that drives the GPT model: AdamW with…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.