Cheat sheets

RLHF, DPO and Preference Tuning

Reward model plus PPO against DPO and its family. Losses, beta and KL settings, failure modes and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

RLHF against DPO#

FactorRLHF (PPO)DPO
Models to train3: SFT, reward, policy1: policy only
Training loops3: SFT, RM, PPO2: SFT, DPO
Hyperparameterslr, KL coeff, clip ratio, RM lr, epochslr, beta, epochs
Reward modelSeparate trainingImplicit in model probabilities
AlgorithmPPO, complex and unstableSupervised learning, stable
Models in memory3-4 during PPO2: current and reference
ComputeThree loopsAbout one-third of RLHF
DataOnline loop with new ratingsFixed preference dataset

From Lesson 10.08

Use X when#

  • DPO: 5,000-20,000 pairs, limited compute, or many dataset experiments.
  • RLHF: large scale, multi-objective rewards, or an iterative online rating loop.
  • SimPO: length bias is evident. It also drops the reference model.
  • KTO: you only have unpaired good or bad labels.
  • ORPO: you want one stage from the base model, with no SFT checkpoint.
  • BPO: DPO logs show degraded chosen log-probs.
  • IPO: preference strengths vary widely and DPO saturates.

From Lesson 10.08, Lesson 18.03, Lesson 09.09

RLHF pipeline#

initializefreezetrainresponserewardKL penaltyupdatemodelSFT modelstage 1storePreference pairsprompt, chosen, rejectedmodelPolicyinit from SFTmodelReward modelscalar head, Bradley-TerrymodelReferencefrozen SFT copyprocessPPO updatereward minus beta KLinitializefreezetrainresponserewardKL penaltyupdatemodelSFT modelstage 1storePreference pairsprompt, chosen,rejectedmodelPolicyinit from SFTmodelReward modelscalar head,Bradley-TerrymodelReferencefrozen SFT copyprocessPPO updatereward minus betaKL

From Lesson 10.07, Lesson 09.09

One DPO epoch (capstone)#

1Referenceunder torch.no_grad(), store chosen and rejectedlog-probs once.2Policysequence_log_prob sums completion log-probs.Prompt tokens add nothing.3Margindiff_w - diff_l, the chosen log-ratio minus therejected log-ratio.4Loss-F.logsigmoid(beta * margin) for each triple.5Stepopt.zero_grad(), loss.backward(), opt.step() onAdam.6Checkrun_demo returns 1 when the margin does not riseor the loss does not drop.

From 19.40 code/main.py, Lesson 19.40

DPO loss and reference freeze#

diff_w = logp_w_pol - logp_w_refchosen log-ratio
diff_l = logp_l_pol - logp_l_refrejected log-ratio
margin = diff_w - diff_limplicit reward gap over beta
z = beta * margin
loss = -F.logsigmoid(z)stable -log sigmoid
policy.load_state_dict(reference.state_dict())start from equal weights
for p in reference.parameters():
p.requires_grad = Falsereference never moves
target = 1.0 / (2.0 * beta) if beta > 0 else 0.0
loss = (margin - target) ** 2IPO variant

From 19.40 code/main.py

Reward model and KL reward (toy)#

p = sigmoid(r_pos - r_neg)Bradley-Terry P(chosen wins)
grad_scale = 1 - p
w[t] += lr * grad_scale * cchosen tokens up, rejected down
kl_term = kl(probs_new, probs_ref)
reward = rm_score - beta * kl_termKL-penalized reward
advs = [(r - mean_r) / sd_r for r in rewards]normalized advantages
ratio = math.exp(logp - log_pi_old)
clipped = (adv > 0 and ratio > 1 + eps) or (adv < 0 and ratio < 1 - eps)

From 09.09 code/main.py

Tasks in code#

TaskCode
Train a Bradley-Terry RMtrain_rm(n_pairs=500, lr=0.1)
Toy PPO-RLHF looprlhf_loop(w, updates=300, beta=0.1)
TRL PPO with adaptive KLPPOConfig(init_kl_coef=0.05, target_kl=6.0)
Train DPO, lesson 10.08dpo_train(..., lr=5e-6, beta=0.1)
Sweep betabetas=[0.01, 0.1, 0.3, 1.0]
Response log-probsequence_log_prob(policy, prompt, chosen)
Compare six lossespython3 code/main.py
Capstone end to endsys.exit(run_demo())

From 09.09 code/main.py, Lesson 09.09, 10.08 code/main.py, 18.03 code/main.py, 19.40 code/main.py

Losses#

RM loss
-\log \sigma(R_\phi(x, y_+) - R_\phi(x, y_-)), the Bradley-Terry pairwise logistic.
Implicit reward
r = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x). Z(x) cancels.
DPO
-\log \sigma(\beta (\log \frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \log \frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}))
IPO
squared error of the log-ratio gap against \frac{1}{2\beta}. The gap stays bounded.
SimPO
length-normalized \frac{\beta}{|y|} \log \pi(y \mid x) with a margin \gamma. No reference.
Gradient sign
\partial L / \partial \log \pi_\theta(y_w) = -\beta (1 - \sigma(z)). Chosen goes up.

From Lesson 09.09, Lesson 10.08, Lesson 18.03, Lesson 19.40

Defaults#

SettingDefaultNote
RLHF KL β0.01-0.05Typical range
InstructGPT KL β0.02lr=1.5e-5, 4 PPO epochs
PPO clip ε0.2Ratio clipped to 1 ± ε
TRL init_kl_coef0.05With target_kl=6.0
DPO β, lesson 10.080.1Sweet spot 0.1 to 0.3
DPO lr, lesson 10.085e-65 epochs
Capstone DPOConfig beta0.2lr 1e-3, 30 epochs
Capstone warmup_epochs8Reference pretrain first

From Lesson 09.09, Lesson 10.07, Lesson 10.08, 19.40 code/main.py

Troubleshooting#

SymptomCauseFix
Reward climbs, human eval flatReward hackingStop early, raise β, broaden RM data
Responses get longerLength hackingLength-normalized reward, or SimPO
Mean KL past ~5-10Policy drifted from SFTAdaptive β that targets a fixed KL
Policy barely changesβ too highLower β
Chosen log-prob falls in DPODegraded Chosen ResponseBPO penalty on the chosen response
DPO gap blows upImplicit reward gap is unboundedIPO, gap capped at 1/(2 beta)
RM scores unreliableRM smaller than the policyRM at least as large as the policy
~30% noisy labelsAmbiguous preferencesAgreement-filtered data, BT temperature

From Lesson 09.09, Lesson 10.07, Lesson 10.08, Lesson 18.03

Questions#

Why pairs, not absolute scores?
People rank two answers well. They score one answer badly.
Why the KL term?
It keeps the policy near the data where the RM was trained.
Why does the RM read the last token?
The causal mask lets it see the whole sequence.
How many forward passes per DPO pair?
Four: two models times two responses.
Do DAAs escape over-optimization?
No. They over-optimize the reference ratio instead.

From Lesson 10.07, Lesson 09.09, Lesson 10.08, Lesson 18.03