Cheat sheets

Reinforcement Learning

MDPs, value methods, policy gradients, PPO, RLHF and GRPO. Formulas, update lines and diagnostics from Phase 9.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

PPO training loop#

actiontransitionvalueadvantageupdateN parallel envsT steps eachActorsoftmax policyRollout buffers, a, r, log_pi_oldCriticv_old per stateGAEadvantages, returnsClipped surrogateK epochs, minibatchesDiagnosticsmean KL, clip fractionactiontransitionvalueadvantageupdateNparallelenvsTstepseachActorsoftmaxpolicyRolloutbuffers, a,r,log_pi_oldCriticv_oldperstateGAEadvantages,returnsClippedsurrogateKepochs,minibatchesDiagnosticsmeanKL,clipfraction

From Lesson 09.08, 09.08 code/main.py, Lesson 09.07

RLHF in three stages#

1SFTfine tune the base model on humandemonstrations.2Preference datacollect rows of prompt, chosen and rejected.3Reward modeltrain a scalar head on the SFT model with theBradley-Terry loss.4Policy and referencestart the policy from SFT. Freeze a copy as thereference.5RewardRM score minus $\beta$ times the KL to thereference.6PPOupdate the token level policy against thatreward.7Monitorwatch mean KL. Past about 5 to 10, drift orreward hacking starts.

From Lesson 09.09

Value formulas#

MDP
the tuple (S, A, P, R, \gamma) with the Markov property.
Return
G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \ldots
Bellman V
V^\pi(s) = \sum_a \pi(a \mid s) \sum_{s', r} P(s', r \mid s, a) [r + \gamma V^\pi(s')]
Value iteration
V(s) \leftarrow \max_a \sum_{s',r} P(s',r \mid s,a) [r + \gamma V(s')]
TD(0)
V(s) \leftarrow V(s) + \alpha [r + \gamma V(s') - V(s)]
Q-learning
Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma \max_{a'} Q(s', a') - Q(s, a)]
SARSA
Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma Q(s', a') - Q(s, a)]
DQN loss
(r + \gamma \max_{a'} Q(s', a'; \theta^-) - Q(s, a; \theta))^2 over replay samples.

From Lesson 09.01, Lesson 09.02, Lesson 09.04, Lesson 09.05

Policy formulas#

Policy gradient
\nabla J(\theta) = \mathbb{E}[ \sum_t G_t \cdot \nabla_\theta \log \pi_\theta(a_t \mid s_t) ]
TD advantage
\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)
GAE
A_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}
PPO ratio
r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)
PPO clip
\mathbb{E}_t [ \min( r_t A_t, \mathop{\mathrm{clip}}(r_t, 1-\epsilon, 1+\epsilon) A_t ) ]
Bradley-Terry
L(\phi) = -\mathbb{E}[ \log \sigma(R_\phi(x, y_+) - R_\phi(x, y_-)) ]
RLHF reward
R_\phi(x, y) - \beta \cdot \mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}})
GRPO advantage
A_i = (r_i - \text{mean}_r) / \text{std}_r over G samples of one prompt.

From Lesson 09.06, Lesson 09.07, Lesson 09.08, Lesson 09.09, Lesson 09.12

Choose a method#

SituationMethod
Small tabular MDP, known modelvalue iteration or policy iteration
Small tabular MDP, samples onlyQ-learning
On-policy, safety criticalSARSA or Expected SARSA
High dimensional state, discrete actionsDQN, with Double DQN in production
Continuous controlPPO with Gaussian policy, or SAC and TD3
LLM alignment from preferencesDPO first, RLHF-PPO with KL penalty
LLM math and code reasoningGRPO with a verifier reward
Two player board gamesAlphaZero or MuZero

From Lesson 09.02, Lesson 09.04, Lesson 09.05, Lesson 09.08, Lesson 09.09, Lesson 09.12

Defaults and numbers#

SettingDefaultNote
Discount \gamma0.99horizon about 1 / (1 - \gamma), so 100 steps
Tabular \alpha and \epsilon0.1Q-learning and SARSA on the 4x4 GridWorld
PPO clip \epsilon0.20.1 is too timid, 0.3 and more is unstable
Value coefficient c_v0.5PPO and A2C
Entropy coefficient c_e0.01PPO and A2C
GAE \lambda0.950 is TD, 1 is MC
RLHF KL \beta0.01-0.05keeps the policy near SFT
GRPO group size G88 to 64 is standard, below 4 is noisy

From Lesson 09.01, Lesson 09.04, Lesson 09.07, Lesson 09.08, Lesson 09.09, Lesson 09.12

Update lines#

TaskCode
Return, reverse sweepG = r + gamma * G
Explore with epsilon greedyif random() < epsilon: return choice(ACTIONS)
Q-learning targettarget = r + (gamma * max(Q[s_next].values()) if not done else 0.0)
SARSA targettarget = r + (gamma * Q[s_next][a_next] if not done else 0.0)
TD stepQ[s][a] += alpha * (target - Q[s][a])
Sync DQN target netif steps % sync_every == 0: target = copy(online)
RLHF reward with KLreward = dot(w, bag([token])) - beta * kl(probs, probs_ref)
GRPO group advantageadvs = [(r - mean_r) / std_r for r in rewards]

From Lesson 09.03, Lesson 09.04, Lesson 09.05, Lesson 09.09, Lesson 09.12

PPO clipped update#

for rec, adv in zip(mb, adv_norm):advantages normalized per minibatch
x = rec["x"]
probs = softmax(logits(theta, x))current policy
logp = math.log(max(probs[rec["a"]], 1e-12))
ratio = math.exp(logp - rec["log_pi_old"])new over old probability
clipped = (adv > 0 and ratio > 1 + eps) or (adv < 0 and ratio < 1 - eps)
pg_scale = 0.0 if clipped else ratio * advclipped means zero gradient
for action in range(N_ACTIONS):
grad_logpi = (1.0 if action == rec["a"] else 0.0) - probs[action]
for j in range(N_FEAT):
theta[action][j] += lr_a * pg_scale * grad_logpi * x[j]
err = rec["ret"] - value(w, x)critic error
for j in range(N_FEAT):
w[j] += lr_v * err * x[j]critic step

From 09.08 code/main.py

Troubleshooting#

SymptomCauseFix
Mean KL past 0.1policy drifts far from the old policyreduce K_EPOCHS or LR
Clip fraction near 0clip never triggersraise LR or K_EPOCHS
Clip fraction 0.5 or moreoverfit to the rolloutlower LR or K_EPOCHS
Reward climbs, human eval dropsreward hacking of the RMstop early, raise \beta, broaden RM data
Responses get longerRM rewards lengthlength normalized reward
Q values too highmax bias on noisy QDouble DQN
DQN divergesdeadly triadkeep the target net and the replay buffer
Policy stops exploringentropy collapseentropy bonus with c_e > 0

From Lesson 09.04, Lesson 09.05, Lesson 09.06, Lesson 09.07, Lesson 09.08, Lesson 09.09

Abbreviations#

MDP
Markov Decision Process. States, actions, transitions, rewards, discount.
TD
Temporal Difference. Update from a one step bootstrapped target.
DQN
Deep Q-Network. Q-learning with replay, target net and reward clipping.
A2C
Advantage Actor-Critic. Synchronous batched parallel envs.
GAE
Generalized Advantage Estimation. Weighted sum of TD residuals.
PPO
Proximal Policy Optimization. Clipped importance ratio, many epochs.
RLHF
Reinforcement Learning from Human Feedback. SFT, then RM, then PPO.
GRPO
Group Relative Policy Optimization. PPO variant without a critic.

From Lesson 09.01, Lesson 09.04, Lesson 09.05, Lesson 09.07, Lesson 09.08, Lesson 09.09, Lesson 09.12

Questions#

On-policy or off-policy?
On-policy learns from its own data. Off-policy learns from another policy's data.
Q-learning or SARSA?
Q-learning bootstraps on the max. SARSA uses the next action it takes and is more cautious.
Why subtract a baseline?
It keeps the gradient unbiased and cuts the variance.
Why the KL in RLHF?
It keeps the policy near the data where the RM was trained.
Why GRPO for reasoning?
No value network, so half the memory. Per prompt normalization fits sparse rewards.

From Lesson 09.03, Lesson 09.04, Lesson 09.06, Lesson 09.09, Lesson 09.12

Tips#

  • Write the five MDP tuples before you write a training loop.
  • Check a tabular agent against the DP value V^{\ast}.
  • Decay \epsilon from 1.0 to 0.01 over the first 10% of training.
  • Normalize advantages per batch to zero mean and unit std.
  • Compute the ratio as exp(log_new minus log_old) for numerical stability.
  • Start PPO with K = 10 epochs and minibatches of 64.

From Lesson 09.01, Lesson 09.02, Lesson 09.05, Lesson 09.07, Lesson 09.08