Papers to code

Binary and multi-class focal loss with gamma and alpha, from logits and without overflow, plus the binary gradient.

From Loss Functions

Normalize image and text embeddings, build the temperature-scaled similarity matrix, and compute the symmetric InfoNCE loss and its gradient.

From Open-Vocabulary Vision — CLIP

GQA forward with H query heads and G shared key and value groups, plus mean pooling of a multi-head checkpoint into groups.

From Multi-Head Attention

Route tokens to the top k experts with renormalized softmax gates, and compute the Switch Transformer auxiliary loss E times sum f_i P_i.

From Mixture of Experts (MoE)

Interpolate along straight paths, regress the velocity x1 - x0 with the flow matching loss, and integrate a velocity field with Euler steps.

From Flow Matching & Rectified Flows

Build the linear beta schedule and alpha bars, sample x_t from x_0 in closed form, and compute the mean and variance of q(x_{t-1} | x_t, x_0).

From Diffusion Models — DDPM from Scratch

Normalize rewards within each group, estimate the KL with k3, and compute the clipped GRPO objective averaged per answer over valid tokens.

From RL for Games — AlphaZero, MuZero, and the LLM-Reasoning Era

The direct preference optimization loss, its implicit rewards and its gradient, from policy and reference log probabilities.

From DPO: Direct Preference Optimization

The min(1, p/q) acceptance rule and the residual distribution that keep speculative decoding exactly equal to the target model.

From Speculative Decoding — Draft, Verify, Repeat

FlashAttention in numpy: a running max and a running sum over key blocks give exact attention for any block size.

From KV Cache, Flash Attention & Inference Optimization