LLM inference

Sweep the temperature on fixed logits, measure entropy and top 1 probability, find where entropy crosses 2 bits, and count the tokens top p keeps.

From GPT — Causal Language Modeling

Forward pass of a linear layer with a low rank adapter, merge of the adapter into the weight, and the trainable parameter count.

From Fine-Tuning with LoRA & QLoRA

Train a table of next-character logits with gradient descent until the held-out negative log-likelihood beats a unigram model.

From Text Generation Before Transformers — N-gram Language Models

Append keys and values to a preallocated cache, attend one query over the valid prefix, and reject an append past the capacity.

From KV Cache, Flash Attention & Inference Optimization

Scale logits by temperature, keep the top k or the top p mass, renormalize, and draw one token with a seeded rng.

From GPT — Causal Language Modeling

Absmax int8 quantization with one scale per tensor and one scale per row, with dequantization and a zero-row rule.

From Quantization: Making Models Fit

Sweep symmetric absmax quantization from 2 to 8 bits on weights with outliers, compare one scale per tensor with one scale per row.

From Quantization: Making Models Fit

Use real SGLang runs on an NVIDIA L4 to find the saturation point, the SLO limit, the FP8 gain and the effect of a smaller prefill chunk.

From Inference Metrics — TTFT, TPOT, ITL, Goodput, P99

Count weighted adjacent pairs, pick the best pair with a fixed tie rule, merge it without overlap and train a BPE merge list.

From BPE Tokenizer From Scratch

Keep the best beam_width partial sequences over a step function, finish beams at EOS, and rank them with a length penalty.

From Sequence-to-Sequence Models

The min(1, p/q) acceptance rule and the residual distribution that keep speculative decoding exactly equal to the target model.

From Speculative Decoding — Draft, Verify, Repeat