Large Corpus Downloader
Training a language model begins long before the first forward pass. The corpus has to land on disk, decompressed, deduplicated, and addressable, with the resume story already worked out before the network drops at 4 percent. This lesson builds a streaming downloader that pulls compressed shards, decompresses on the fly with Zstandard, fingerprints near-duplicates via MinHash plus locality-sensitive hashing, and writes a shard manifest the rest of the pipeline can trust. Stream remote shards with urllib and decompress with zstandard without buffering the whole file in memory. Resume partial downloads by issuing HTTP Range requests against a verified byte offset. Build a MinHash signature per document and bucket it with LSH so near-duplicates collide. Emit a shard manifest with content hash, byte size, document count, and dedup verdict. The first time you train on a 200 GB corpus the network drops at percent 41 and the script exits with a urllib exception. The second time it drops at percent 78. By percent 99 you have rewritten the loop three times. The two failures you have to design for from minute one are partial-download resume and duplicate document removal. Both have well-known solutions; both are routinely skipped because the pipeline begins as a one-line requests.get call that grew teeth. Resume is an HTTP problem. The server has to honour Range, the…
Large Corpus Downloader: Training a language model begins long before the first forward pass. The corpus has to land on disk, decompressed, deduplicated, and…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.