Cheat sheets

Value Methods against Policy Methods

Q-learning, SARSA, DQN, REINFORCE, A2C and PPO side by side. Targets, defaults, diagnostics and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Four methods side by side#

FactorQ-learning, SARSADQNREINFORCEA2C, PPO
LearnsQ tableNeural Q(s, a; \theta)Policy \pi_\theta(a \mid s)Actor \pi_\theta plus critic V_\phi(s)
Picks actionsε-greedy on Qε-greedy on QSample from \pi_\thetaSample from \pi_\theta
DataQ-learning off-policy, SARSA on-policyOff-policy, replay bufferOn-policy, one updateA2C one update, PPO K epochs
Targetr + \gamma \max_{a'} Q(s', a')Same, from target net \theta^-Return G_tGAE advantage
Bias and varianceBootstrap bias, low varianceMax bias upwardUnbiased, high varianceCritic bias, far lower variance
Continuous actionsNoNo, route to SAC or TD3Yes, Gaussian policyYes, Gaussian policy
Toy GridWorld~2,000 episodes~500 episodes~500 episodes with a baselinePPO with ε 0.2, K 4

From Lesson 09.04, Lesson 09.05, Lesson 09.06, Lesson 09.07, Lesson 09.08

Value side: TD targets#

a = epsilon_greedy(Q, s, rng, epsilon)behaviour action
s_next, r, done = step(s, a)
best_next = max(Q[s_next].values())Q-learning: greedy max
target = r + gamma * best_next
a_next = epsilon_greedy(Q, s_next, rng, epsilon)
target = r + gamma * Q[s_next][a_next]SARSA: the next action taken
Q[s][a] += alpha * (target - Q[s][a])same TD update for both
Q[s][a] += alpha * (r - Q[s][a])terminal step, no bootstrap

From 09.04 code/main.py

Policy side: gradient, GAE, clip#

G = r + gamma * Greward-to-go, walked backwards
grad = (1.0 if i == a else 0.0) - probs[i]onehot minus probs
theta[i][j] += lr * adv * grad * x[j]REINFORCE ascent
delta = traj[t]["r"] + gamma * next_v - traj[t]["v"]TD residual
gae = delta + gamma * lam * gaeGAE with lam=0.95
ratio = math.exp(logp - rec["log_pi_old"])PPO importance ratio
clipped = (adv > 0 and ratio > 1 + eps) or (adv < 0 and ratio < 1 - eps)
pg_scale = 0.0clipped sample, no gradient

From 09.06 code/main.py, 09.07 code/main.py, 09.08 code/main.py

Defaults#

SettingDefaultNote
Tabular α and ε0.1Pin α in [0.05, 0.3]
Discount γ0.99Default in the lesson code
DQN target sync10,000Atari env steps
GAE λ0.95The 2026 default
Value weight c_v0.5A2C and PPO
Entropy weight c_e0.01A2C and PPO
PPO clip ε0.20.1 is timid, 0.3+ is unstable
PPO epochs K10Minibatches of 64

From Lesson 09.04, Lesson 09.05, Lesson 09.07, Lesson 09.08

Troubleshooting#

SymptomCauseFix
Q values overestimatemax over noisy Q is biased upDouble DQN or Double Q-learning
DQN divergesDeadly triadKeep the target net and the replay buffer
DQN overfits earlyGradients on ~20 samplesWait for a few thousand transitions
Policy stops exploringEntropy collapseEntropy bonus, c_e above 0
Policy gradient explodesHuge returnsNormalize G to ~N(0, 1) per batch
Actor learns noiseCritic is still randomWarm up the critic, slow actor LR
PPO mean KL past 0.1Too many epochs or high LRReduce K_EPOCHS or LR
Clip fraction near 0Clip never triggersRaise LR or K_EPOCHS

From Lesson 09.04, Lesson 09.05, Lesson 09.06, Lesson 09.07, Lesson 09.08

Use X when#

  • Q-learning: small tabular environments. It learns the optimal policy directly.
  • SARSA or Expected SARSA: on-policy safety-critical tasks. They stay conservative during exploration.
  • DQN: high-dimensional state with discrete actions. It is the reference off-policy algorithm.
  • REINFORCE: read it, rarely run it. loss = -advantage * log_prob is REINFORCE with a baseline.
  • A2C: batched parallel envs on a GPU. GPU-based A2C dominates A3C in 2026.
  • PPO: the default for MuJoCo, Atari and RLHF for LLMs.
  • SAC or TD3: continuous control. DQN has no policy network.
  • GRPO: reasoning LLMs. It is a PPO variant without a critic.

From Lesson 09.04, Lesson 09.05, Lesson 09.06, Lesson 09.07, Lesson 09.08

DQN training loop#

transitionsminibatch 32ε-greedy actiony = r + γ maxq[a]clone every 200 stepsclientEnvironment4x4 GridWorldstoreReplay buffercapacity 2000modelOnline netweights thetamodelTarget netfrozen copyprocesstrain_stepsquared TD errortransitionsminibatch 32ε-greedy actiony = r + γ maxq[a]clone every 200 stepsclientEnvironment4x4 GridWorldstoreReplay buffercapacity 2000modelOnline netweights thetamodelTarget netfrozen copyprocesstrain_stepsquared TDerror

From Lesson 09.05, 09.05 code/main.py

PPO training loop#

1Rolloutcollect N times T transitions across N parallelenvs.2Snapshotstore log_pi_old for each action at rollouttime.3GAEcompute advantages, freeze them, normalize perbatch.4Epochsfor K epochs, shuffle the buffer intominibatches.5Ratioratio = exp(logp - rec["log_pi_old"]).6Clipa sample past 1 ± ε on the beneficial side getszero gradient.7Lossesadd value MSE and the entropy bonus. Take thegradient step.8Discarddrop the rollout and collect a new one.

From Lesson 09.08, 09.08 code/main.py

Tasks in code#

TaskCode
Tabular SARSAsarsa(episodes, alpha=0.1, gamma=0.99)
Tabular Q-learningq_learning(episodes, alpha=0.1, gamma=0.99)
One DQN gradient steptrain_step(online, target, mb, gamma, lr)
Sync the target nettarget = clone(online)
REINFORCE with baselinereinforce(episodes, use_baseline=True)
GAE advantagesgae_advantages(traj, gamma=0.99, lam=0.95)
Train A2Cactor_critic(episodes, lam=0.95)
PPO update with diagnosticsmean_kl, clip_frac = ppo_update(...)

From 09.04 code/main.py, 09.05 code/main.py, 09.06 code/main.py, 09.07 code/main.py, 09.08 code/main.py

Key formulas#

TD error
\delta = r + \gamma V(s') - V(s)
SARSA target
r + \gamma Q(s', a') with the action taken next.
DQN loss
(r + \gamma \max_{a'} Q(s', a'; \theta^-) - Q(s, a; \theta))^2
Double DQN
r + \gamma Q(s', \arg\max_{a'} Q(s', a'; \theta); \theta^-)
Policy gradient
\nabla J = \mathbb{E}[G_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)]
Advantage
A = G - \hat{V}(s). Same expectation, lower variance.
GAE
A_t = \sum_l (\gamma \lambda)^l \delta_{t+l}. λ 0 is TD, λ 1 is MC.
PPO clip
\min(r_t A_t, \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon) A_t)

From Lesson 09.04, Lesson 09.05, Lesson 09.06, Lesson 09.07, Lesson 09.08

Abbreviations#

TD
temporal difference. Update after every step from a bootstrapped target.
SARSA
the tuple (s, a, r, s', a'). On-policy TD.
GLIE
Greedy in the Limit with Infinite Exploration. The ε condition.
DQN
Q-learning with a neural Q, replay buffer and target network.
GAE
Generalized Advantage Estimation. Weighted sum of n-step advantages.
A2C, A3C
synchronous batched, or asynchronous threaded, actor-critic.
TRPO
Trust Region Policy Optimization. KL constraint, conjugate gradients.
PPO
Proximal Policy Optimization. Clipped importance ratio.

From Lesson 09.04, Lesson 09.05, Lesson 09.07, Lesson 09.08

Questions#

Why is Q-learning off-policy?
The max assumes the greedy policy, whatever action the agent takes.
Why is SARSA safer on the cliff?
It puts exploration noise into its Q values.
Why are value methods weak on continuous actions?
argmax over a 10-dimensional torque is hard.
Why is a baseline unbiased?
The expected baseline times the log-policy gradient is 0.
Why can PPO reuse a rollout?
The clip stops the update once the policy drifts too far.
Most common PPO bug?
A flipped sign. Minimize the negative of the clipped surrogate.

From Lesson 09.04, Lesson 09.06, Lesson 09.08, Lesson 09.07