Practice track

RL basics

Returns, advantages and the PPO objective. 6 problems, about 135 minutes. Do them in order: each problem uses ideas from the problems before it.

0 / 6 solved

  1. 01
    Discounted returns with episode ends

    Reward-to-go returns for one episode and for a batch of joined episodes with done flags, plus return normalization.

    From Policy Gradient — REINFORCE from Scratch

    ImplementEasy15 min
  2. 02
    Q-learning and SARSA updates

    One tabular Q-learning or SARSA update with no bootstrap at a terminal step, and epsilon-greedy actions from a seeded rng with lowest-index ties.

    From Temporal Difference — Q-Learning & SARSA

    ImplementEasy15 min
  3. 03
    Generalized advantage estimation

    TD errors and GAE advantages with gamma, lambda, done flags and a bootstrap value, plus the critic targets.

    From Actor-Critic — A2C and A3C

    ImplementMedium25 min
  4. 04
    PPO clipped surrogate loss

    The PPO loss to minimize from new and old log probabilities and advantages, the clip fraction, and the gradient with respect to the new log probabilities.

    From Proximal Policy Optimization (PPO)

    ImplementMedium25 min
  5. 05
    GRPO group advantages and clipped objective

    Normalize rewards within each group, estimate the KL with k3, and compute the clipped GRPO objective averaged per answer over valid tokens.

    From RL for Games — AlphaZero, MuZero, and the LLM-Reasoning Era

    ResearchMedium30 min
  6. 06
    DPO loss and implicit rewards

    The direct preference optimization loss, its implicit rewards and its gradient, from policy and reference log probabilities.

    From DPO: Direct Preference Optimization

    ResearchMedium25 min