Practice track

Optimizers

Gradient descent, momentum and Adam, one update rule at a time. 4 problems, about 85 minutes. Do them in order: each problem uses ideas from the problems before it.

0 / 4 solved

  1. 01
    Linear regression by gradient descent

    Mean squared error, its gradients for weights and bias, and a batch gradient descent loop that records the cost after each update.

    From Linear Regression

    ImplementEasy20 min
  2. 02
    Sweep the learning rate

    Sweep gradient descent learning rates on a seeded regression, find the best rate and the first rate that diverges, and compare with 2 over the top eigenvalue.

    From Learning Rate Schedules and Warmup

    LabEasy20 min
  3. 03
    SGD with momentum

    One momentum update with the lesson's velocity rule, and a loop that minimizes a quadratic with or without Nesterov lookahead.

    From Optimizers

    ImplementEasy20 min
  4. 04
    Adam and AdamW step

    One Adam update with bias correction and a step counter that starts at 1, plus AdamW decoupled weight decay, as a pure function of the state.

    From Optimizers

    ImplementMedium25 min