Experiment labs

Sweep gradient descent learning rates on a seeded regression, find the best rate and the first rate that diverges, and compare with 2 over the top eigenvalue.

From Learning Rate Schedules and Warmup

Sweep the temperature on fixed logits, measure entropy and top 1 probability, find where entropy crosses 2 bits, and count the tokens top p keeps.

From GPT — Causal Language Modeling

Fit polynomials of degree 1 to 12 on resampled training sets and measure bias squared, variance and test error at each degree.

From Bias-Variance Tradeoff

Sweep symmetric absmax quantization from 2 to 8 bits on weights with outliers, compare one scale per tensor with one scale per row.

From Quantization: Making Models Fit

Use real SGLang runs on an NVIDIA L4 to find the saturation point, the SLO limit, the FP8 gain and the effect of a smaller prefill chunk.

From Inference Metrics — TTFT, TPOT, ITL, Goodput, P99