Phase 09: Reinforcement Learning

Proximal Policy Optimization (PPO)

A2C throws away each rollout after one update. PPO wraps the policy gradient in a clipped importance ratio so you can do 10+ epochs on the same data without the policy exploding. Schulman et al. (2017). Still the default policy-gradient algorithm in 2026. A2C (Lesson 07) is on-policy: the gradient E{πθ}[A · ∇ log πθ] requires data sampled from the current πθ. Take one update, and πθ changes; the data you used is now off-policy. Re-use it and your gradient is biased. Rollouts are expensive. On Atari, one rollout across 8 envs × 128 steps = 1024 transitions and a dozen seconds of environment time. Throwing that away after one gradient step is wasteful. Trust Region Policy Optimization (TRPO, Schulman 2015) was the first fix: constrain each update so the KL divergence between old and new policy stays below δ. Theoretically clean, but requires a conjugate-gradient solve per update. Nobody runs TRPO in 2026. PPO (Schulman et al. 2017) replaces the hard trust-region constraint with a simple clipped objective. One extra line of code. Ten epochs per rollout. No conjugate gradients. Good-enough theoretical guarantees. Nine years later it is still the default policy-gradient algorithm for everything from MuJoCo to RLHF. PPO clipped surrogate objective: ratio clipping at 1 ± ε The importance ratio. rt(θ) = πθ(at | st) / π{θold}(at…

Proximal Policy Optimization (PPO): A2C throws away each rollout after one update. PPO wraps the policy gradient in a clipped importance ratio so you can do…

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.