Tokenized Dataset with Sliding Window
A pretraining run is a function from token ids to gradients. This lesson builds the conveyor that feeds the ids in. Convert a raw corpus into a stream of token ids by calling the tokenizer once. Slice the id stream into fixed-length windows with a configurable overlap stride. Build a PyTorch Dataset that returns input and target tensors for next-token prediction. Wrap the dataset in a DataLoader with a deterministic shuffle seeded per epoch. Reason about the trade-off between stride, redundancy, and effective dataset size. A pretraining run reads one batch of token ids at a time and updates the model. The shape of each batch is fixed by the training contract. For a causal language model, the batch holds (B, T) input ids and (B, T) target ids where the target is the input shifted left by one. The job of the data pipeline is to produce that contract on demand, in a deterministic and reproducible way, from a corpus that may be several gigabytes of raw text. This lesson builds the pipeline. The tokenizer from the previous lesson turns text into a long flat list of ids. A sliding window slices that list into training examples. A custom Dataset exposes the examples as tensors. A DataLoader batches them and shuffles them with a known seed. A causal LM…
Tokenized Dataset with Sliding Window: A pretraining run is a function from token ids to gradients. This lesson builds the conveyor that feeds the ids in.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.