Cheat sheets

Neural Network Basics

Perceptron, forward pass, backprop, loss choice and a mini framework. Formulas, shapes, defaults and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Forward and backward through a 2-layer net#

forwarda1a2backwardclientInput xno computationprocessHidden layera1 = sigmoid(W1 x +b1)processOutput layera2 = sigmoid(W2 a1+ b2)storeLossL = (a2 - y)^2dL/dz22(a2 - y) a2 (1 -a2)dL/dW2 = dL/dz2a1dL/da1 = dL/dz2 W2dL/dz1dL/da1 a1 (1 - a1)dL/dW1 = dL/dz1 xdL/db1 = dL/dz1forwarda1a2backwardclientInput xno computationprocessHidden layera1 = sigmoid(W1 x +b1)processOutput layera2 = sigmoid(W2 a1 +b2)storeLossL = (a2 - y)^2dL/dz22(a2 - y) a2 (1 -a2)dL/dW2 = dL/dz2a1dL/da1 = dL/dz2 W2dL/dz1dL/da1 a1 (1 - a1)dL/dW1 = dL/dz1 xdL/db1 = dL/dz1

From Lesson 03.03, Lesson 03.02

Value node with backward#

out = Value(self.data + other.data, (self, other), '+')
self.grad += out.gradadd passes the gradient through
other.grad += out.grad
out = Value(self.data * other.data, (self, other), '*')
self.grad += other.data * out.gradmul swaps the inputs
other.grad += self.data * out.grad
self.grad += (s * (1 - s)) * out.grad # sigmoid reuses s
build_topo(self)topological sort of the graph
self.grad = 1.0dL/dL = 1
for v in reversed(topo):
v._backward()push gradients to children

From Lesson 03.03, 03.03 code/main.py

Mini framework training loop#

model = Sequential(Linear(2, 16), ReLU(), Linear(16, 16), ReLU(),
Linear(16, 8), ReLU(), Linear(8, 1), Sigmoid())
criterion = BCELoss()
optimizer = Adam(model.parameters(), lr=0.01)
model.train()dropout and batch stats on
for batch_inputs, batch_targets in loader:
for x, t in zip(batch_inputs, batch_targets):
pred = model.forward(x)
loss = criterion(pred, t)
optimizer.zero_grad()
grad = criterion.backward()dL/dpred
model.backward(grad)modules in reverse
optimizer.step()
model.eval()running stats, no dropout

From Lesson 03.10, 03.10 code/main.py

Defaults in the lesson code#

SettingDefaultNote
Perceptron learning_rate0.1epochs=100
Adam lr0.001beta1=0.9, beta2=0.999, eps=1e-8
SGD lr0.01param -= lr * grad
Dropout p0.5Survivors scale by 1/(1-p)
BatchNormmomentum=0.1eps=1e-5, running stats for eval
BCE clip eps1e-15BCELoss in the framework uses 1e-7
alpha0.1Label smoothing, target 0.91 for 10 classes
temperature0.07Contrastive loss, SimCLR default

From Lesson 03.01, Lesson 03.05, Lesson 03.10

Troubleshooting#

SymptomCauseFix
Perceptron never converges on XORXOR is not linearly separableAdd a hidden layer
Classifier outputs 0.5 for allMSE loss on classificationUse cross-entropy
Early layers do not learnSigmoid shrinks gradients per layerReLU (Lesson 04), scaled init
Loss is inf or NaN\log(0) = -\inftyClip p to [eps, 1 - eps]
math.exp(1000) overflowLarge input to sigmoidClamp to [-500, 500]
Gradient wrong for a reused value= overwrites other pathsAccumulate with +=
Gradients grow each stepOld gradients stayCall zero_grad() first
Test results change each runDropout still onCall model.eval()

From Lesson 03.01, Lesson 03.02, Lesson 03.03, Lesson 03.05, Lesson 03.10

One training step#

1DataLoaderyields a batch of inputs and targets. Shuffleseach epoch.2Modelruns the forward pass layer by layer. Each layerstores its input and output.3Losscompares predictions to targets and gives onescalar.4Optimizeroptimizer.zero_grad() resets every gradient tozero.5Lossloss.backward() sends gradients right to leftthrough the graph.6Optimizeroptimizer.step() updates each parameter, for SGDparam -= lr * grad.7Modelmodel.eval() before test. Inference runs onlythe forward pass.

From Lesson 03.10, Lesson 03.03

Formulas#

Step
\text{step}(z) = 1 if z \ge 0, else 0.
Perceptron rule
w_i = w_i + learning_rate * error * x_i, with error = y_true minus y_pred.
Sigmoid
\sigma(z) = \frac{1}{1 + e^{-z}}, range (0, 1). Its derivative is a(1 - a).
Layer
z = W x + b, then a = \text{sigmoid}(z).
MSE
\frac{1}{n} \sum (y_{\text{pred}} - y_{\text{true}})^2, gradient \frac{2}{n}(y_{\text{pred}} - y_{\text{true}}).
BCE
-\left(y \log(p) + (1 - y) \log(1 - p)\right)
CCE
-\sum_i y_i \log(p_i). Only the true class contributes.
Softmax plus CE gradient
probability minus 1 for the true class, probability for the others.

From Lesson 03.01, Lesson 03.02, Lesson 03.03, Lesson 03.05

Tasks in code#

TaskCode
Train a perceptron on ANDp_and = Perceptron(2) then p_and.train(and_data)
Chain layersNetwork([hidden, output])
Autograd net, 2-4-1net = Network([2, 4, 1])
Backprop from the losstotal_loss.backward()
Stable multi-class lossF.cross_entropy(logits, labels)
Smoothed targetslabel_smoothing=0.1
Pick an optimizerAdam(model.parameters(), lr=0.01)
Batch the dataDataLoader(train_data, batch_size=16, shuffle=True)

From Lesson 03.01, Lesson 03.02, Lesson 03.03, Lesson 03.05, Lesson 03.10

Which loss#

CommandWhat it does
F.mse_lossRegression. Penalizes outliers quadratically
MAE / HuberRegression that is robust to outliers
F.binary_cross_entropyBinary classification on probabilities
F.cross_entropyMulti-class. Log-softmax and NLL in one step
Label smoothingAdd when multi-class CE is overconfident
Focal LossImbalanced classes. Down-weights easy examples
Triplet LossAnchor, positive, negative with a margin
InfoNCELarge batch self-supervised embeddings

From Lesson 03.05

Numbers to remember#

  • The sigmoid derivative peaks at 0.25 when a = 0.5.
  • Ten sigmoid layers scale the gradient by at most 0.25^{10} = 0.000001.
  • MSE that predicts 0.5 on a 50/50 binary task gives 0.25 and learns nothing.
  • Cross-entropy gives -\log(0.5) = 0.693 and -\log(0.99) = 0.01.
  • A wrong confident guess p = 0.01 costs -\log(0.01) = 4.6.
  • Random guess on 10 classes costs -\log(0.1) = 2.3.
  • Focal loss with \gamma = 2 weights an easy example at 0.01, a hard one at 0.81.
  • One backward pass gives all gradients. Nudging each weight needs one forward pass per weight.

From Lesson 03.03, Lesson 03.05, Lesson 03.02

Questions#

What shape is W?
Rows match the current layer. Columns match the previous layer.
Why sigmoid, not step, for training?
Sigmoid is smooth, so gradients exist.
Why store activations?
The backward pass needs them. Memory buys one pass.
Why topological sort?
A node gets all its gradient before it passes it on.
Why F.cross_entropy, not softmax then log?
One operation is numerically stable.
Why train() and eval()?
Dropout and BatchNorm act differently in each mode.
Why is zero_grad a separate call?
Gradients accumulate with += until you reset them.

From Lesson 03.01, Lesson 03.02, Lesson 03.03, Lesson 03.05, Lesson 03.10