Deep learning core

Inverted dropout with a given random generator: training forward with an exact mask, backward with the same mask, identity at inference.

From Regularization

Binary and multi-class focal loss with gamma and alpha, from logits and without overflow, plus the binary gradient.

From Loss Functions

Batched softmax, mean cross-entropy and its gradient, without overflow for large logits.

From Loss Functions

Train a small sigmoid network with full-batch gradient descent until it labels noisy XOR points with 95 percent held-out accuracy.

From Backpropagation from Scratch

LayerNorm with gamma and beta and RMSNorm with a weight, both over the last axis, with eps inside the square root.

From The Full Transformer — Encoder + Decoder

One Adam update with bias correction and a step counter that starts at 1, plus AdamW decoupled weight decay, as a pure function of the state.

From Optimizers

Train a 1-16-1 tanh network on noisy samples of sin(x) until its held-out mean squared error against the true curve is below 0.01.

From Activation Functions

Train a 2-24-24-1 tanh network with mini-batch Adam until it separates two noisy spirals with 95 percent held-out accuracy.

From Build Your Own Mini Framework

Normalize image and text embeddings, build the temperature-scaled similarity matrix, and compute the symmetric InfoNCE loss and its gradient.

From Open-Vocabulary Vision — CLIP

Interpolate along straight paths, regress the velocity x1 - x0 with the flow matching loss, and integrate a velocity field with Euler steps.

From Flow Matching & Rectified Flows

Build the linear beta schedule and alpha bars, sample x_t from x_0 in closed form, and compute the mean and variance of q(x_{t-1} | x_t, x_0).

From Diffusion Models — DDPM from Scratch

Forward pass with a cache and a backward pass that returns the gradient of every weight and bias of a two-layer sigmoid network.

From Backpropagation from Scratch

Training and inference forward passes with running statistics, and the full backward pass for x, gamma and beta.

From Regularization