Information Theory and Numerical Stability

Entropy, cross-entropy and KL, stable softmax and log-sum-exp, float formats, PCA and distance choice.

Information formulas#

Surprise: $inline$ . Base 2 gives bits. Base e gives nats.
Entropy: $inline$ . Fair coin 1 bit. A 99% coin 0.08 bits.
Cross-entropy: $inline$ . A one-hot label gives $inline$ .
KL divergence: $inline$ . It is not symmetric.
Mutual information: $inline$ . Zero means independent.
Conditional entropy: $inline$ . Decision tree splits minimize it.
Perplexity: $inline$ in nats. Perplexity 50 is a uniform pick from 50 tokens.
Label smoothing: $inline$ . With 0.1 and 4 classes the true class gets 0.925.
Units: 1 nat is 1.4427 bits. PyTorch and TensorFlow use nats.

def logsumexp_stable(values):

c = max(values)largest exponent becomes 0

return c + math.log(sum(math.exp(v - c) for v in values))

def log_softmax_stable(logits):

c = max(logits)

lse = c + math.log(sum(math.exp(z - c) for z in logits))

return [z - lse for z in logits]log probs, no overflow

def cross_entropy_stable(true_class, logits):

log_probs = log_softmax_stable(logits)

return -log_probs[true_class]never calls log(0)

Task	Code
Entropy in bits	`entropy(probs, base=2)`
KL divergence	`kl_divergence(p, q, base=2)`
Perplexity from mean CE	`perplexity(avg_cross_entropy, base="e")`
Safe softmax	`softmax_stable(logits)`
Numerical gradient	`numerical_gradient(f, x, h=1e-5)`
Compare to backprop	`check_gradient(analytical, numerical)`
Clip the gradient norm	`clip_by_norm(gradients, max_norm)`
Variance with a large mean	`welford_variance(values)`

Setting	Default	Note
Gradient check `h`	1e-5	Typical range 1e-5 to 1e-7
Gradient check pass	1e-7	Above 1e-3 something is wrong
`max_norm`	1.0	Transformers. 0.5 for RL, 5.0 for simple nets
LayerNorm `epsilon`	1e-5	Prevents division by zero
Epsilon in log and divide	1e-8	`log(x + 1e-8)`
t-SNE perplexity	5-50	The lesson uses 30
UMAP `n_neighbors`	15	With `min_dist` 0.1
PCA cumulative variance	0.95	Or a 90-95% threshold

Symptom	Cause	Fix
Loss is NaN after a few epochs	Logits too large, softmax overflowed	Stable softmax, lower LR, clip gradients
Loss stuck at log(num_classes)	Near-uniform outputs, no learning	Check labels, loss and dead ReLUs
Val accuracy 1-3% low	Mixed precision without loss scaling	Dynamic loss scaling or bfloat16
Gradient norm 0.0 in some layers	Dead ReLUs or float16 underflow	LeakyReLU or GELU, gradient scaling
Results differ between GPUs	Float addition is not associative	`torch.use_deterministic_algorithms(True)`
`exp()` returns inf in the loss	Raw logits without max subtraction	`torch.nn.functional.log_softmax()`
Diverges after float16 switch	Grads below 6e-8, values above 65,504	AMP with loss scaling, or bfloat16
Wrong variance for large values	Catastrophic cancellation	`welford_variance` or center the data

Command	What it does
`cosine_similarity(a, b)`	Text and embeddings. Ignores magnitude
`dot_product(a, b)`	Recommendation. Magnitude encodes popularity
`l2_distance(a, b)`	Pixels and spatial data at comparable scale
`l1_distance(a, b)`	Sparse high-dim features. Robust to outliers
`linf_distance(a, b)`	Worst-case deviation in one dimension
`mahalanobis_distance(x, y, cov_matrix)`	Outliers with correlated features
`jaccard_similarity(set_a, set_b)`	Sets and tags. IoU for segmentation
`edit_distance(s1, s2)`	Strings. Insert, delete or substitute

Curse of dimensionality: at 1000 dimensions the max/min distance ratio is ~1.02.
PCA: center, covariance, np.linalg.eigh, sort by eigenvalue, keep the top k.
Explained ratio: one eigenvalue divided by the sum of all eigenvalues.
Pick k: keep 90-95% of variance, find the elbow, or sweep k on the task.
Reconstruction error: the sum of the dropped eigenvalues. High error marks an outlier.
t-SNE: keeps local neighbors. Stochastic and $inline$ . Cluster gaps mean nothing.
UMAP: faster than t-SNE. It keeps more global structure.
Kernel PCA: PCA in a kernel feature space. RBF separates concentric circles.

Why subtract the max?: The largest term becomes exp(0) = 1. The sum is at least 1.
Why bfloat16 for training?: It has the float32 range. Loss scaling is usually unnecessary.
Why minimize cross-entropy?: It equals negative log-likelihood. H(P) is constant.
Why is KL not a distance?: It is not symmetric and fails the triangle inequality.
Why Wasserstein for GANs?: It gives gradients when distributions do not overlap.
Why does L1 give sparse weights?: Loss contours touch the L1 diamond at a corner.
Why centered differences?: The error is $inline$ . Forward differences give $inline$ .

Do not compare floats with ==. Use math.isclose().
Check for nan and inf after each forward pass while you debug: math.isfinite(x).
Clamp inputs to exp: exp(clamp(x, -80, 80)).
Clip by norm with torch.nn.utils.clip_grad_norm_(). It keeps the gradient direction.
Check each new layer or loss with torch.autograd.gradcheck().
With L2-normalized embeddings, cosine and dot product give the same result.