Pre-Training Data Pipelines

Clean, dedup, download, tokenize, shard and window a pretraining corpus. Python from the lessons.

Tasks#

Task	Code
Run a lesson demo	`python3 code/main.py`
Run the tests	`python3 -m unittest discover code/tests -v`
Drop near duplicates	`deduplicate(filtered_docs, threshold=0.8)`
Pack ids into sequences	`pack_sequences(token_ids, seq_length, pad_id=0)`
Download one shard	`process_shard(plan, downloader, dedup, manifest)`
Write HDF5 shards	`pipeline.write_corpus(shards)`
Check shard hashes	`validate_corpus(entries)`
Read a token range	`store.get_slice(start, stop)`

def clean_text(text):

text = re.sub(r"<[^>]+>", "", text)strip HTML tags

text = re.sub(r"http\S+", "", text)drop URLs

text = re.sub(r"[^\x20-\x7E\n]", "", text)printable ASCII only

text = re.sub(r"\n{3,}", "\n\n", text)collapse blank lines

text = re.sub(r" {2,}", " ", text)collapse spaces

return text.strip()

def quality_filter(text, min_words=50, max_ratio_caps=0.3, max_ratio_special=0.1):

...False for stubs, ALL CAPS spam, symbol noise

ds = SlidingWindowDataset(ids, context_length=16, stride=8)stride None means context_length

SlidingWindowDataset.count_windows(num_ids, context_length, stride)1 + usable // stride

x, y = ds[0]window[:-1], window[1:]

loader = make_dataloader(ds, batch_size=4, base_seed=7, epoch=0)

generator.manual_seed(base_seed + epoch)same seed, same order

Setting	Default	Note
`min_words`	50	Stub pages fail
`max_ratio_caps`	0.3	ALL CAPS spam fails
`max_ratio_special`	0.1	Symbol noise fails
`threshold`	0.8	Jaccard to drop a copy
`num_hashes`	128	Signature length
`bands`	16	Lesson 03. Lesson 42 uses `DEFAULT_BANDS = 32`
`DEFAULT_SHINGLE_WIDTH`	5	Demo uses `shingle_width=3`
`CHUNK_BYTES`	`1 << 16`	Download read size

Setting	Default	Note
`DEFAULT_CHUNK_SIZE`	8192	Demo uses `chunk_size=512`
`DEFAULT_WINDOW_SIZE`	64	Reads `window_size + 1` tokens
`BOUNDARY_TOKEN_ID`	0	Separator between documents
`TOKEN_DTYPE`	`np.uint16`	Byte ids offset by one
`maxshape`	`(None,)`	Resizable dataset
`libver`	`"latest"`	Required on the writer for SWMR

Window count: max(0, 1 + (N - (T + 1)) // S) for N ids, context T, stride S.
Stride: T // 2 doubles the dataset. 1 grows it by a factor of T.
LSH collision: 1 - (1 - s^r)^b with k = b * r.
LSH threshold: near (1/b)^(1/r). For b=32, r=4 it is near s = 0.42.
Compression: English is about 3-4 characters per token.
Sequence utilization: below 90% means packing wastes compute.
Dedup yield: the Llama team removed approximately 38% of web data.
Chinchilla: scale parameters and tokens roughly equally.

Symptom	Cause	Fix
Resumed shard holds garbage	Offset and file differ	Check the sha256 prefix first
Server ignores `Range`	Status is not 206	Close it and do a full GET
Duplicated suffix after power loss	Bytes written before checkpoint	Checkpoint first, then write
Model learns to predict zero	Reader reads zero padding	Truncate at `token_count`
"file is locked"	No `libver="latest"` or late SWMR	Create datasets, then set `swmr_mode`
Throughput halves	Chunk does not match window	Chunk a multiple of `window_size`
Runs diverge for no reason	No shuffle seed	Pass `base_seed` per epoch
`ValueError` from `LSHIndex`	`bands` does not divide `num_hashes`	Pick bands that divide 128

Why pack instead of pad?: Padding wastes compute on tokens that teach nothing.
Why a block-diagonal mask?: Doc A tokens must not attend to Doc B tokens.
What does the slicer do with a short tail?: It drops the last window.
Why dedup before tokenization?: Tokenization is expensive. Do not pay it twice.
Why tombstone duplicates?: The manifest keeps the link to the keeper.
Why one HDF5 per source shard?: A crash loses at most one shard.