Cheat sheets

Alignment Training: Instructions, DPO Family, RLAIF

Pick and tune RLHF, DPO, IPO, KTO, SimPO, ORPO, BPO and Constitutional AI. Losses, defaults and failures.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

RLHF vs RLAIF vs DPO#

FactorRLHF (PPO)RLAIF (CAI)DPO
Preference signalHuman labelers rank pairsFeedback model reads principlesHuman or AI prefs
Reward modelRequired (separate training)Trained on AI preferencesImplicit in model probabilities
RL algorithmPPO (complex, unstable)PPO against the RMSupervised learning (stable)
Models to train3 (SFT + reward + policy)Same as RLHF1 (policy only)
Models in memory3-4Same as RLHF2 (current + reference)
Hyperparameterslr, KL coeff, clip ratio, RM lr, epochs x3Same as RLHFlr, beta, epochs
Online loopYes, retrain the RM on new ratingsYes, uses the iterative loopNo, fixed dataset of pairs
Main failureReward hackingGoodhart persists, sycophancy dropsOver-optimizes the reference ratio

From Lesson 10.08, Lesson 18.05, Lesson 18.01, Lesson 18.03

DPO family losses#

FactorIPOKTOSimPOORPO
DataPairsUnpaired good or bad labelsPairsPairs
Reference policyYesYesNoNo
Fixes in DPOUnbounded reward gapNeeds paired dataLength biasSeparate SFT stage
Loss shapeSquared error, target 1/(2 beta)Prospect theory, loss aversionLength-normalized log-prob, margin gammaNLL plus lambda times odds ratio
Toy functiontrain_dpo(pairs, variant="ipo")train_kto(labels)train_simpo(pairs)train_orpo(pairs)
Toy defaultsbeta: float = 0.1beta: float = 0.1beta: float = 1.5, gamma: float = 0.5lam: float = 0.1

From Lesson 18.03, 18.03 code/main.py, Lesson 10.08

Two pairwise losses in code#

def bradley_terry_loss(reward_preferred, reward_rejected):
diff = reward_preferred - reward_rejectedRM score gap
loss = -np.log(sigmoid(diff) + 1e-8)stage 2 RM loss
def dpo_loss(policy_logprob_preferred, policy_logprob_rejected,
ref_logprob_preferred, ref_logprob_rejected, beta=0.1):
preferred_ratio = policy_logprob_preferred - ref_logprob_preferred
rejected_ratio = policy_logprob_rejected - ref_logprob_rejected
logit = beta * (preferred_ratio - rejected_ratio)4 log-probs in
loss = -np.log(sigmoid(logit) + 1e-8)no reward model
preferred_reward = beta * preferred_ratioimplicit reward
rejected_reward = beta * rejected_ratio
"reward_margin": float(preferred_reward - rejected_reward)must grow

From 10.07 code/main.py, 10.08 code/main.py

Troubleshooting#

SymptomCauseFix
Reward climbs, outputs degeneratebeta too low, RM blind spotsRaise the KL penalty
No gain over SFTbeta too highLower beta
Benchmarks drop after RLHFAlignment taxPPO-ptx: mix in pretraining gradients
Longer and longer responsesRM scores lengthAdversarial RM data, or SimPO
Model agrees with everythingSycophancy from annotatorsSeveral reward models, adversarial data
Chosen log-prob falls in DPODegraded Chosen ResponseSwitch to BPO
DPO reward gap blows upImplicit gap is unboundedUse IPO, gap capped by 1/(2 beta)
Gold reward peaks, then collapsesDAAs over-optimize tooBetter data, ensembles, early stopping

From Lesson 18.01, Lesson 18.03, Lesson 10.07

Use X when#

  • DPO: you have 5,000-20,000 pairs or limited compute. It needs about one third of the RLHF compute.
  • SimPO: length bias is evident in DPO.
  • KTO: you have unpaired binary feedback.
  • ORPO: you want a single-stage pipeline from a base model.
  • BPO: DPO logs show degraded chosen log-probs.
  • IPO: preference strengths vary widely and DPO saturates.
  • RLHF: "better" has many dimensions, or you rate new samples in an online loop.
  • RLAIF: you replace the human labeler with a model that reads explicit principles.

From Lesson 18.03, Lesson 10.08, Lesson 18.05

InstructGPT pipeline#

SFTgenerateranktrainfreezeinitresponserewardKL penaltyupdatemodelBase modelcompletes textmodelSFT modelcross-entropy onpairsstorePreferencepairsprompt, winner,losermodelReward modelscalar head,Bradley-TerrymodelReference modelfrozen SFTmodelPolicy modelbeing optimizedprocessPPO updatereward - beta *KLclientHumanannotatorsrank KcompletionsSFTgenerateranktrainfreezeinitresponserewardKL penaltyupdatemodelBase modelcompletes textmodelSFT modelcross-entropy on pairsstorePreference pairsprompt, winner, losermodelReward modelscalar head,Bradley-TerrymodelReference modelfrozen SFTmodelPolicy modelbeing optimizedprocessPPO updatereward - beta * KLclientHuman annotatorsrank K completions

From Lesson 18.01, Lesson 10.07

Constitutional AI pipeline#

1SFT modelhelpful but not yet harmless. It answers ared-team prompt.2Critique modelreads a sampled principle and critiques theresponse.3Revision steprevises the response. The revised response isthe SFT target.4Feedback modelranks pairs of completions against sampledprinciples.5Reward modeltrains on the AI-generated preferences.6PPOoptimizes the policy against the RM, as inInstructGPT.7Constitutional Classifiergates outputs. v1 had 23.7% compute overhead, v2~1%.

From Lesson 18.05

Loss formulas#

Bradley-Terry RM
L_{\text{RM}} = -\log \sigma(r(x, y_w) - r(x, y_l))
RLHF objective
\mathbb{E}[r(x, y)] - \beta \, \mathrm{KL}(\pi \,\|\, \pi_{\text{SFT}})
PPO-ptx
J(\pi) + \gamma \, \mathbb{E}_{x \sim D_{\text{pretrain}}}[\log \pi(x)]
Implicit reward
\beta \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)}
DPO
-\log \sigma(\beta (\log \frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \log \frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}))
IPO
square of (log-ratio gap - \frac{1}{2\beta}). The margin stays bounded.
SimPO
-\log \sigma(\frac{\beta}{|y_w|} \log \pi(y_w) - \frac{\beta}{|y_l|} \log \pi(y_l) - \gamma)
ORPO
L_{\text{NLL}}(y_w) + \lambda L_{\text{OR}}, no reference policy.

From Lesson 18.01, Lesson 18.03, Lesson 10.07, Lesson 10.08

Toy pipelines in code#

TaskCode
Run a lesson demopython3 code/main.py
Standard toy RLHFstage3_ppo(sft, rm, beta=0.1)
Show reward hackingstage3_ppo(sft, rm, beta=0.0)
Inject a reward model bugstage2_reward_model(bias=[0.5, 0.0, 0.0])
Train DPO, IPO or BPOtrain_dpo(pairs, variant="bpo")
Build the CAI SFT corpusbuild_cai_sft_corpus(500)
AI labeler ranks a pairai_feedback_rank(a, b)
Sweep DPO betabetas=[0.01, 0.1, 0.3, 1.0]

From 18.01 code/main.py, 18.03 code/main.py, 18.05 code/main.py, 10.07 code/main.py, 10.08 code/main.py

Defaults#

SettingDefaultNote
PPO kl_coeff0.02InstructGPT. Llama 2 Chat used 0.01
PPO lr1.5e-5InstructGPT: 256K episodes, 4 PPO epochs per batch
PPO clip epsilon0.2Typical value
DPO beta0.1Sweet spot 0.1 to 0.3
DPO lr5e-6num_epochs=5 in dpo_train
RM lr1e-4num_epochs=10 in train_reward_model
Toy stage3_pposteps: int = 300batch: int = 32, lr: float = 0.1
KTO loss aversion2.0Toy weight on bad labels. Lesson 08 says 1.5x

From Lesson 10.07, 10.07 code/main.py, Lesson 10.08, 10.08 code/main.py, 18.01 code/main.py, 18.03 code/main.py

Questions#

Why did 1.3B InstructGPT beat 175B GPT-3?
Labelers preferred it about 70% of the time. Alignment is a different axis from capability.
Can RLHF add facts?
No. The base model sets the capability floor.
Why pairwise labels?
Humans compare two responses well. They assign absolute scores badly.
What RM accuracy is good?
About 72% on held-out pairs. Inter-annotator agreement was about 73%.
Is RLAIF only a cheaper RLHF?
No. The signal is legible: you read the principle, critique and revision.
Do DAAs escape Goodhart?
No. Over-optimization moves to the reference policy ratio.

From Lesson 18.01, Lesson 10.07, Lesson 10.08, Lesson 18.05, Lesson 18.03