Phase 09: Reinforcement Learning

Monte Carlo Methods — Learning from Complete Episodes

Dynamic programming needs a model. Monte Carlo needs nothing but episodes. Run the policy, watch the returns, average them. The simplest idea in RL — and the one that unlocks everything downstream. Dynamic programming is elegant, but it assumes you can query P(s' | s, a) for every state and action. Almost nothing in the real world works that way. A robot cannot analytically compute the distribution over camera pixels after a joint torque. A pricing algorithm cannot integrate over every possible customer reaction. An LLM cannot enumerate all possible continuations after a token. You need a method that only needs the ability to sample from the environment. Run the policy. Get a trajectory s0, a0, r1, s1, a1, r2, …, sT. Use it to estimate values. That is Monte Carlo. The shift from DP to MC is philosophically important: we move from known model + exact backup to sampled rollouts + averaged return. The variance jumps, but the applicability explodes. Every RL algorithm after this lesson — TD, Q-learning, REINFORCE, PPO, GRPO — is a Monte Carlo estimator at heart, sometimes with bootstrapping layered on top. Monte Carlo: rollout, compute returns, average; first-visit vs every-visit The core idea, in one line: V^π(s) = Eπ[Gt | st = s] ≈ (1/N) Σi G^{(i)}(s) where G^{(i)}(s) are observed returns following visits…

Monte Carlo Methods — Learning from Complete Episodes: Dynamic programming needs a model. Monte Carlo needs nothing but episodes. Run the policy, watch the…

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.