Phase 09: Reinforcement Learning

Policy Gradient — REINFORCE from Scratch

Stop estimating value. Parameterize the policy directly, compute the gradient of expected return, step uphill. Williams (1992) wrote it in one theorem. It is why PPO, GRPO, and every LLM RL loop exist. Q-learning and DQN parameterize the value function. You pick actions by argmax Q. That is fine for discrete actions and discrete states. It breaks when actions are continuous (which argmax over a 10-dimensional torque?) or when you want a stochastic policy (argmax is deterministic by construction). Policy gradients parameterize the policy instead. πθ(a | s) is a neural net that outputs a distribution over actions. Sample from it to act. Compute the gradient of expected return with respect to θ. Step uphill. No argmax. No Bellman recursion. Just gradient ascent on J(θ) = E{πθ}[G]. The REINFORCE theorem (Williams 1992) tells you this gradient is computable: ∇J(θ) = Eπ[ G · ∇θ log πθ(a | s) ]. Run an episode. Compute the return. Multiply by ∇ log πθ(a | s) at every step. Average. Gradient-ascent. Done. Every LLM-RL algorithm in 2026 — PPO, DPO, GRPO — is a refinement of REINFORCE. Understanding it in your fingers is the prerequisite for the rest of this phase, and for Phase 10 · 07 (RLHF implementation) and Phase 10 · 08 (DPO). Policy gradient: softmax policy, log-π gradient, return-weighted update The…

Policy Gradient — REINFORCE from Scratch: Stop estimating value. Parameterize the policy directly, compute the gradient of expected return, step uphill.…

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.