Data Pipelines for Pre-Training
The model is a mirror. It reflects whatever data you feed it. Feed it garbage, it reflects garbage with perfect fluency. Build a streaming data pipeline that tokenizes, chunks, shuffles, and batches terabytes of text without loading it all into memory. Implement data quality filters (deduplication, language detection, content filtering) used in real pre-training pipelines. Create fixed-length training sequences with proper attention masks and document boundary handling. Profile pipeline throughput to ensure the dataloader keeps up with GPU training speed. You have a tokenizer. Now you need data. Not a dataset. Not a CSV file. Terabytes of text -- cleaned, deduplicated, filtered for quality, tokenized into fixed-length sequences, and served in randomized batches fast enough that your 8-GPU cluster never waits for the next batch. Most people think training an LLM is about the model architecture. It is not. Llama 3 used 15.6 trillion tokens. GPT-3 used 300 billion. DeepSeek-V2 used 8.1 trillion. The architecture across all three is roughly the same: stacked transformer blocks with attention and feedforward layers. The difference in output quality comes overwhelmingly from the data. The Chinchilla paper from DeepMind made this precise. For a given compute budget, there is an optimal ratio of model parameters to training tokens. Chinchilla showed that most models in 2022 were dramatically undertrained -- they had too many parameters…
Data Pipelines for Pre-Training: The model is a mirror. It reflects whatever data you feed it. Feed it garbage, it reflects garbage with perfect fluency.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.