Cheat sheets

Math for Machine Learning

Gradients, probability, information, optimizers, decompositions and float safety. NumPy, SciPy and PyTorch.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Computational graph, forward and backward#

acbackwardx1 = 2dy/dx1 = 3x2 = 3dy/dx2 = 2Multiplya = 6b = 1dy/db = 1Addc = 7reluy = 7Seeddy/dy = 1acbackwardx1 = 2dy/dx1 = 3x2 = 3dy/dx2 = 2Multiplya = 6b = 1dy/db = 1Addc = 7reluy = 7Seeddy/dy = 1

From Lesson 01.05

Key formulas#

Chain rule
\frac{dy}{dx} = f'(g(x)) \cdot g'(x)
Bayes
P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}
Entropy
H(P) = -\sum_{x} p(x) \log(p(x))
Cross-entropy
H(P, Q) = -\sum_{x} p(x) \log(q(x))
KL divergence
D_{KL}(P \,\|\, Q) = H(P, Q) - H(P). Not symmetric.
Log-sum-exp
\log \sum_i e^{x_i} = \max(x) + \log \sum_i e^{x_i - \max(x)}
Cosine similarity
\frac{a \cdot b}{\lVert a \rVert_2 \, \lVert b \rVert_2}, from -1 to +1
SVD
A = U \Sigma V^T. Rank k: A_k = U_k \Sigma_k V_k^T

From Lesson 01.05, Lesson 01.07, Lesson 01.09, Lesson 01.13, Lesson 01.14, Lesson 01.11

Update rules#

Gradient descent
w = w - \text{lr} \cdot \text{gradient}
Momentum
v = \beta v + \text{gradient}, then w = w - \text{lr} \cdot v
Adam moments
m = \beta_1 m + (1 - \beta_1) g, v = \beta_2 v + (1 - \beta_2) g^2
Bias correction
\hat{m} = m / (1 - \beta_1^t), \hat{v} = v / (1 - \beta_2^t)
Adam step
w = w - \text{lr} \cdot \hat{m} / (\sqrt{\hat{v}} + \epsilon)
Cosine schedule
\text{lr}_{\min} + 0.5 (\text{lr}_{\max} - \text{lr}_{\min})(1 + \cos(\pi t / T))
Newton step
w_{\text{new}} = w_{\text{old}} - H^{-1} \cdot \text{gradient}
Temperature
p_i = e^{z_i / T} / \sum_j e^{z_j / T}

From Lesson 01.08, Lesson 01.04, Lesson 01.16

Tasks in NumPy#

TaskCode
Dense layernp.maximum(0, weights @ inputs + bias)
Solve a square systemnp.linalg.solve(A, b)
Least squaresnp.linalg.lstsq(X, y, rcond=None)[0]
Thin SVDU, S, Vt = np.linalg.svd(A, full_matrices=False)
Rank k approximationU[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
Pseudoinverse solvenp.linalg.pinv(A) @ b
Row normsnp.linalg.norm(X, axis=1, keepdims=True)
Batched matmulnp.einsum("bij,bjk->bik", batch_A, batch_B)

From Lesson 01.02, Lesson 01.11, Lesson 01.17, Lesson 01.14, Lesson 01.12, Lesson 01.01

Tasks in PyTorch and SciPy#

TaskCode
AdamW for transformerstorch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)
Cosine scheduletorch.optim.lr_scheduler.CosineAnnealingLR(adam, T_max=100)
Clip by normtorch.nn.utils.clip_grad_norm_()
Check a backward passtorch.autograd.gradcheck()
Stable log of softmaxtorch.nn.functional.log_softmax()
Inverse CDF of N(0, 1)stats.norm(loc=0, scale=1).ppf(0.975)
PCA to 2DSklearnPCA(n_components=2).fit_transform(X)
t-SNE to 2DTSNE(n_components=2, perplexity=30, random_state=42)

From Lesson 01.08, Lesson 01.13, Lesson 01.16, Lesson 01.10

Autograd in a few lines#

x = torch.tensor(2.0, requires_grad=True)record operations on x
y = x ** 2 + 3 * x + 1builds the graph
y.backward()reverse mode pass
print(x.grad)7.0 = 2*x + 3
def __mul__(self, other):
out = Value(self.data * other.data, (self, other), '*')
def _backward():
self.grad += other.data * out.gradlocal grad times upstream
other.grad += self.data * out.grad+= sums over all uses
out._backward = _backward
return out
self.grad = 1.0seed dy/dy = 1
for v in reversed(topo):output to inputs
v._backward()chain rule at each node

From Lesson 01.05

Stable softmax and cross-entropy#

def softmax_stable(logits):
max_logit = max(logits)subtract the max first
exps = [math.exp(z - max_logit) for z in logits]largest term is 1
total = sum(exps)
return [e / total for e in exps]
def cross_entropy_stable(true_class, logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
log_sum_exp = math.log(sum(math.exp(s) for s in shifted))
log_prob = shifted[true_class] - log_sum_expstay in log space
return -log_prob

From Lesson 01.13

PCA with SVD#

1Centersubtract the mean from each feature.2Covariance$C = \frac{1}{n-1} X^T X$ for centered X.3Decomposecall np.linalg.svd(A) on the data. Avoid eig of$A^T A$.4Componentsthe right singular vectors V are the principalcomponents.5Variancecomponent i explains $\sigma_i^2 / (n-1)$.6Choose kkeep components up to 90 to 95% cumulativevariance.7Projectkeep the top k directions and drop the rest.

From Lesson 01.10, Lesson 01.11

Defaults and numbers#

SettingDefaultNote
Adam lr0.001start here for most problems
Adam beta20.999beta1 is 0.9
Adam epsilon1e-8avoids division by zero
SGD momentum lr0.01with momentum 0.9
Gradient check h1e-51e-5 to 1e-7 is typical
Transformer max_norm1.00.5 for RL, 5.0 for simpler nets
Dynamic loss scale start65536halve on inf, double after N clean steps
Top-p0.9with temperature 0.7 for general use

From Lesson 01.08, Lesson 01.13, Lesson 01.16

Troubleshooting#

SymptomCauseFix
Loss is NaN after a few epochssoftmax overflow or lr too highstable softmax, lower lr, clip gradients
Loss stuck at log(num_classes)near uniform outputscheck labels, loss function, dead ReLUs
Accuracy 1 to 3% low with float16gradient underflowdynamic loss scaling or bfloat16
Gradient norm 0.0 in some layersdead ReLU or float16 underflowLeakyReLU or GELU, gradient scaling
Results differ across GPUsfloat addition is not associativetorch.use_deterministic_algorithms(True)
0.1 + 0.2 == 0.3 is False0.1 is not exact in binarymath.isclose() or a tolerance
Solution changes a lot for small input changeslarge condition numberadd \lambda I or drop collinear features
One unseen word zeroes Naive BayesMLE gives probability zeroLaplace smoothing, add 1 to counts

From Lesson 01.13, Lesson 01.17, Lesson 01.07, Lesson 01.08

Questions#

Why reverse mode?
Many inputs and one loss. One backward pass gives all gradients.
Why cross-entropy?
Minimizing it maximizes the likelihood of the data.
Why bfloat16 for training?
It has the float32 range, so values above 65,504 do not overflow.
Why SVD and not the eigenvectors of A.T @ A?
That product squares the condition number.
Why the reparameterization trick?
z = \mu + \sigma \epsilon lets gradients flow through \mu and \sigma.

From Lesson 01.05, Lesson 01.09, Lesson 01.13, Lesson 01.11, Lesson 01.16

Tips#

  • Check shapes: (m x n) @ (n x p) = (m x p).
  • Sum log probabilities. Do not multiply many small probabilities.
  • Never compare floats with ==.
  • Check gradients when you write a new layer or loss.
  • Perplexity is the exponential of cross-entropy in nats.
  • In a gradient check, a relative error below 1e-5 is acceptable.

From Lesson 01.02, Lesson 01.13, Lesson 01.07, Lesson 01.09, Lesson 01.08