Cheat sheets

Scaling Laws and Pre-Training

Chinchilla math, over-training, the data pipeline from raw text to packed batches, and a mini GPT training loop.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Data pipeline#

sequences, masksstoreRaw textCommon Crawl~250 TB rawprocessHTML stripvisible textonlyprocessLanguagedetectionlid.176.bin, 0.8confidenceprocessQualityfilterperplexity orheuristicsprocessDeduplicationMinHash + LSHprocessPII removalregex and NERprocessTokenize +packEOS between docsqueueDataLoadershuffled batchessequences, masksstoreRaw textCommon Crawl ~250 TBrawprocessHTML stripvisible text onlyprocessLanguage detectionlid.176.bin, 0.8confidenceprocessQuality filterperplexity orheuristicsprocessDeduplicationMinHash + LSHprocessPII removalregex and NERprocessTokenize + packEOS between docsqueueDataLoadershuffled batches

From Lesson 10.03

Formulas#

Compute
C \approx 6 \times N \times D FLOPs for a standard transformer.
Hoffmann loss
L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E
Fitted constants
\alpha \approx 0.34, \beta \approx 0.28, E \approx 1.69, A \approx 406, B \approx 411.
Compute-optimal
N_{\text{opt}} \sim C^{0.5}, D_{\text{opt}} \sim C^{0.5}, D/N \approx 20.
Jaccard
shared shingles over all shingles. Remove a copy above 0.8.
Compression ratio
characters per token. English is about 3-4.
Start loss
about \ln(V). A 256-token byte vocab starts at 5.55.
Attention scale
divide by \sqrt{d_k}. For 64 dims that is 8.

From Lesson 07.13, Lesson 10.03, Lesson 10.04

Models against Chinchilla#

FactorGPT-3ChinchillaLlama 3 8B
Parameters175B70B8B
Tokens~300B1.4T15T
Tokens per paramAbout 1.7About 201,875
VerdictUndertrainedCompute-optimal by design94× past optimal
WhyKaplan laws: push N hardMinimize loss per FLOPCheap to serve at scale

From Lesson 07.13, Lesson 10.03

Compute-optimal against over-trained#

FactorChinchilla-optimalOver-trained
MinimizesTraining loss per training FLOPTotal cost with inference
Tokens per paramAbout 20100 to 500, by serving volume
Cost paidOnce, at trainingExtra pretrain FLOPs once
Inference costHigher, N is largerLower, N is smaller
Code checkcompute_optimal(C)chinchilla_loss(N_under, D_over)
ExampleChinchilla 70B on 1.4TLlama 3 8B on 15T

From Lesson 07.13, 07.13 code/main.py

Run the data pipeline#

raw_docs = generate_sample_corpus()
cleaned_docs = [clean_text(doc) for doc in raw_docs]strip HTML, URLs, spaces
filtered_docs = [doc for doc in cleaned_docs if quality_filter(doc)]length, caps, symbols
deduped_docs, num_removed = deduplicate(filtered_docs, threshold=0.8)MinHash + LSH
tokenizer = SimpleTokenizer()
tokenizer.train_bpe(all_text, num_merges=100)
token_ids = tokenize_corpus(deduped_docs, tokenizer)adds eos_id after each doc
sequences, masks = pack_sequences(token_ids, seq_length, pad_id=0)seq_length 128
loader = PreTrainingDataLoader(sequences, masks, batch_size)batch_size 4, shuffled
stats = compute_statistics(deduped_docs, token_ids, sequences, tokenizer.vocab_size())

From 10.03 code/main.py

Functions in the lessons#

TaskCode
Predict the losschinchilla_loss(N, D)
Find the optimal N and Dcompute_optimal(C_flops, n_grid=200)
Word shinglesget_shingles(text, k=5)
Bucket a signaturelsh_buckets(signature, bands=16)
Train the tiny GPTtrain_mini_gpt(corpus, num_steps=200)
GPT-2 family countsparameter_breakdown()
Weights and KV memorymemory_estimate()

From 07.13 code/main.py, 10.03 code/main.py, 10.04 code/main.py, Lesson 10.04

One training step#

1Batchtake seq_len + 1 tokens. Inputs drop the lasttoken. Targets drop the first.2Masknp.triu(np.full((seq_len, seq_len), -1e9), k=1)blocks future positions.3Forwardembeddings, all blocks, final LayerNorm. Logitsuse token_embed.T.4Losscross_entropy_loss(logits, target_ids) with amax-subtracted log-softmax.5Backwardcompute gradients for all parameters.6Updateblock.ffn.W1 -= lr * gW1 and the same for eachtensor.7ScheduleGPT-2 warms up over 2,000 steps, then cosinedecay.

From Lesson 10.04, 10.04 code/main.py

Defaults#

SettingDefaultNote
min_words50Drops stub pages
max_ratio_caps0.3Drops ALL CAPS spam
max_ratio_special0.1Drops symbol noise
num_hashes128MinHash signature length
bands16More bands give more recall
threshold0.8Jaccard cut for a duplicate
vocab_size50257MiniGPT with 768 dims, 12 heads, 12 layers
max_seq_len1024GPT-2 context window

From 10.03 code/main.py, 10.04 code/main.py

Troubleshooting#

SymptomCauseFix
Model repeats passages verbatimDuplicates in the dataMinHash dedup, about 38% of web data
Output has HTML tagsRaw web textStrip HTML before training
Docs leak into each otherPacked docs share attentionUse a block-diagonal mask
Sequence utilization below 90%Padding wastePack docs with EOS between them
English score below 0.8 confidenceNot clean EnglishFilter the document out
Model diverges earlyHigh learning rate at startWarm up the learning rate
exp overflows to infinityLarge logitsSubtract the max before exp
Gradients vanish near block 1No residual pathAdd x + sublayer(x)

From Lesson 10.03, Lesson 10.04

GPT-2 Small numbers#

Total
124,438,272 parameters with weight tying.
Token embeddings
(50257, 768) is 38,597,376.
Per block
7,080,960. Attention 2,359,296 and FFN 4,718,592.
Heads
12 heads of 64 dims. FFN goes 768 to 3,072.
Weight tying
the head reuses the token embedding. Saves 38M.
KV cache
18,432 values per token. About 75MB at 1024 tokens in FP32.
Init
weights use a 0.02 standard deviation.

From Lesson 10.04

Use scaling laws to decide#

  • Below 20 tokens per base-model param, expect a fine-tune to saturate.
  • When inference takes most of the budget, pick a smaller model trained longer.
  • Beyond 1000× Chinchilla-optimal, log-loss changes become noise.
  • Plan budgets against continuous loss. Benchmark jumps are often scorer artifacts.
  • Better data, Muon and MoE shift the constants, not the shape.
  • Expect a data ceiling: ~5 to 10 trillion high-quality English tokens.
  • Llama 3 mix: 50% web, 25% code, 13% books and papers, 8% math, 4% multilingual.

From Lesson 07.13, Lesson 10.03

Questions#

What is E?
The irreducible loss. The entropy of the data itself.
Why over-train?
You pay training cost once and inference cost forever.
Why not pad each doc?
Padding wastes compute. Pack docs and mask.
Why is decode slow?
It is memory-bound. Prefill is compute-bound.

From Lesson 07.13, Lesson 10.03, Lesson 10.04