RLHF: reward models and PPO

Try to write down a reward function for “a helpful answer”. It has to work for a question about tax law, a request for a limerick, a bug report and a medical worry, and it has to return a number. You can’t write it. You can’t write the perfect answer to every prompt either, which is what supervised fine-tuning needs. But show someone two answers to the same question and ask “which is better?”, and they can usually tell you in a few seconds.

Reinforcement learning from human feedback (RLHF) turns that one ability into a training signal. Collect many “this one is better” judgments, fit a model that predicts them, and use that model’s score as the reward for reinforcement learning. Chapter 11 ended with the objective “maximize reward, but stay close to the reference model” and left the reward as a black box $r(x, y)$. This chapter opens the box: where $r$ comes from (a reward model fitted to comparisons), and how the objective gets optimized (PPO from Chapter 9, run one token at a time). This is the recipe behind InstructGPT and the first ChatGPT.

The recipe in one picture

RLHF as practiced from 2020 to about 2024 has three steps. Each uses a different kind of human data.

Step 1  Supervised fine-tuning (SFT)
        humans write good answers (x, y*)  ──>  fine-tune the pretrained LM  ──>  π_SFT

Step 2  Reward model (RM)
        prompts x ──> sample K answers from π_SFT ──> humans rank them ──> pairs (x, y_w, y_l)
        fit a scalar model r_φ(x, y) so that r_φ(x, y_w) > r_φ(x, y_l) as often as humans say

Step 3  Reinforcement learning (PPO)
        π_θ ← π_SFT,  π_ref ← π_SFT (frozen)
        repeat: sample y ~ π_θ(·|x), reward = r_φ(x, y) − β·KL to π_ref, PPO update

Each step fixes a weakness of the one before. SFT is behavior cloning (Chapter 10): it gets the model into the right style, but it can only copy what demonstrators wrote, and writing good demonstrations is slow. The reward model needs only judgments, which are much cheaper, and once trained it can score millions of answers that no human will ever read. RL then optimizes the model’s own samples against that score, so the model learns from its own mistakes rather than from someone else’s successes.

InstructGPT, the paper that made this recipe famous, used about 13,000 prompts for SFT, 33,000 for the reward model and 31,000 for PPO, labeled by a team of about 40 contractors (Ouyang et al., 2022).

Five papers that built it

ChatGPT followed in November 2022. OpenAI’s announcement says it was trained with RLHF “using the same methods as InstructGPT, but with slight differences in the data collection setup”.

Collecting preferences

Why comparisons and not scores

The obvious way to get a reward is to ask people for one: “rate this answer from 1 to 7”. In practice this works badly. Your 5 is my 6. My 5 at 9 a.m. is my 6 at 5 p.m., after I’ve seen a hundred bad answers. A score depends on an internal scale that differs between people and drifts over time, and the reward model would have to learn all of that noise.

A comparison cancels the scale. Whatever personal offset you add to every score, it cancels when you only say which of two answers is better. Ziegler et al. (2019) note that labelers found it hard to give consistent fine-grained numbers, and that on synthetic tasks comparisons were almost as informative as absolute scores. Comparisons are also easier: judging is faster than writing, and much faster than calibrating.

Rankings: more pairs per prompt read

The expensive part of labeling is reading the prompt and understanding what a good answer would look like. Once that’s done, comparing more answers is cheap. So instead of one pair, show the labeler $K$ answers and ask for a ranking. A ranking of $K$ answers contains $\binom{K}{2} = K(K-1)/2$ pairwise comparisons: 6 for $K = 4$, 36 for $K = 9$. InstructGPT’s labelers ranked between 4 and 9 answers per prompt.

There is a catch. The 36 pairs from one ranking are strongly correlated: each answer appears in 8 of them. InstructGPT reports that when they shuffled all pairs into one big dataset, the reward model overfit within a single pass. Their fix was to put all $\binom{K}{2}$ pairs from one prompt into the same batch element, which also needs only $K$ forward passes instead of $2\binom{K}{2}$.

Labelers disagree, a lot

How often do two careful people agree on which answer is better? For InstructGPT, the training labelers agreed with each other 72.6 ± 1.5% of the time; held-out labelers 77.3 ± 1.3%. In Stiennon et al.’s summarization work, researchers agreed with each other 73 ± 4% of the time. Roughly one comparison in four goes the other way depending on who you ask.

That number shapes everything that follows. A preference label is not a fact about the two answers; it is a noisy sample. So we need a probabilistic model of how preferences arise, and a reward model that predicts probabilities, not verdicts.

The Bradley–Terry model

A worked example first

Suppose three answers to one prompt have rewards $r_A = 2$, $r_B = 1$, $r_C = -1$. The Bradley–Terry model says the probability that a labeler prefers one answer to another depends only on the difference of their rewards, through the logistic sigmoid $\sigma(z) = 1/(1+e^{-z})$:

A reward gap of 1 means “preferred about 73% of the time” (by coincidence, about the inter-labeler agreement from the last section). A gap of 3 means “almost always”. Add 10 to all three rewards and nothing in the table changes.

The model

For a prompt $x$ and two responses $y_1, y_2$:

Here $\succ$ reads “is preferred to”. The second form is the same thing (divide top and bottom by $e^{r(x,y_1)}$), and it is the more illuminating one: each response gets a weight $e^{r}$, and the chance of winning is your share of the total weight. Bradley and Terry proposed it in 1952, in statistics, for ranking items from paired comparisons.

Where it comes from: noisy utilities

Why this formula and not some other S-shaped curve? Here is one clean story. When a labeler looks at an answer, they form a momentary impression $u = r + \varepsilon$: the answer’s true quality $r$ plus noise $\varepsilon$ (attention, mood, which details they happened to notice). They prefer whichever answer has the larger $u$. If the noise terms are independent and follow a standard Gumbel distribution (the distribution that maxima of many independent variables tend to follow; here it is simply a convenient modeling choice), then

exactly. The derivation is short.

Three ways to read the formula

• Chess ratings. The Elo system predicts that a player rated $R_A$ beats one rated $R_B$ with probability $1/(1 + 10^{-(R_A - R_B)/400})$. That is Bradley–Terry with $r = R \ln 10 / 400$. One unit of reward is about 174 Elo points. Christiano et al. (2017) make this analogy themselves.
• Logistic regression. If you only look at the difference $\Delta = r(x, y_1) - r(x, y_2)$, Bradley–Terry is logistic regression of “was $y_1$ preferred?” on $\Delta$, with the slope fixed to 1.
• A two-state Boltzmann system. Treat each answer as a state with energy $E = -r$ at temperature $T = 1$. The probability of finding the system in state 1 is $e^{-E_1}/(e^{-E_1} + e^{-E_2})$, the Bradley–Terry probability. With $K$ answers it is the canonical ensemble. Two facts carry over directly. Energies are defined up to a constant, and so are rewards. And the temperature sets the scale: noisier labelers act like a higher temperature, so the same quality differences produce probabilities closer to 1/2. Since the model fixes $T = 1$, a reward model fitted to noisy labelers comes out with smaller reward gaps. Reward units are “log-odds of being preferred”, not an absolute measure of quality.

The demo lets you feel the model. Drag the three answers along the reward axis and watch the pairwise probabilities move along the sigmoid; then use the shift buttons.

What Bradley–Terry cannot express

The model assumes every answer has one number, so preferences must be transitive in probability: if A usually beats B and B usually beats C, then A beats C even more often. Real people are not always like that, and a population of labelers with different tastes can produce cycles (A ≻ B ≻ C ≻ A) even when each individual is consistent. Bradley–Terry averages such disagreements into one scalar. Some later work drops the scalar reward entirely and learns a pairwise preference model, then looks for a policy that wins such comparisons against any opponent (for example, Nash learning from human feedback, Munos et al. 2023). Almost all deployed RLHF has used a scalar reward model, so that is what we build.

Training a reward model

Architecture: a language model with a one-number head

A reward model $r_\phi(x, y)$ reads a prompt and a response and outputs one real number. To judge text it has to understand text, so it starts from a pretrained language model, usually the SFT model itself:

tokens:     [ prompt x ............ ][ response y ...... ][EOS]
                         │  transformer (initialized from π_SFT)
hidden:      h_1  h_2  ...                         ...   h_last ∈ R^d
                                                            │  linear head  w·h + b   (replaces the
                                                            ▼                          vocabulary head)
                                                       r_φ(x, y)  ∈ R

The language model’s final layer, which maps the hidden state to logits over the vocabulary, is removed and replaced by a linear layer with one output. The score is read at the last token (usually the end-of-sequence token). In a causal transformer only the last position has attended to the whole prompt and the whole response, so it is the natural place to summarize them. InstructGPT used one 6B-parameter reward model for policies of every size (1.3B, 6B and 175B); they report that 175B reward models trained unstably.

The loss

Fitting Bradley–Terry by maximum likelihood gives the reward model loss: for each labeled pair $(x, y_w, y_l)$, where $y_w$ won and $y_l$ lost,

This is binary cross-entropy where the “logit” is the reward difference and the label is always “the first one won”. Its gradient tells you what training does to each pair. Write $\Delta = r_\phi(x, y_w) - r_\phi(x, y_l)$. Since $\frac{d}{d\Delta}\big[-\log\sigma(\Delta)\big] = -(1 - \sigma(\Delta))$,

Every step raises the winner’s score and lowers the loser’s, weighted by $1 - \sigma(\Delta)$, the probability the model currently assigns to the wrong outcome:

Pairs the model already gets right with confidence stop mattering, so the effort goes to the pairs it gets wrong. And a pair that labelers disagree on (the same two answers labeled once each way) pulls in both directions until $\Delta = 0$: the reward model learns the labelers’ rate of preference, not a verdict (Exercise 12.4).

The reward is only defined up to a constant per prompt

The two responses in a pair always share the prompt $x$. So if you add any function of the prompt alone to the reward,

every difference $r'(x, y_w) - r'(x, y_l)$ is unchanged and so is the loss. The data cannot tell $r_\phi$ from $r'$. This is the “only differences are measurable” idea again, now with a separate zero point for every prompt.

Three consequences:

1. Scores across prompts are not comparable. A reward of 2.0 on one prompt and 0.5 on another doesn’t mean the first answer is better; the two prompts may just have different offsets.
2. RL doesn’t care, in principle. A per-prompt constant changes neither the optimal KL-regularized policy (it cancels in the normalization of $\pi^*$; Exercise 12.3) nor the expected policy gradient (it acts like a baseline, Chapter 7). It does add variance, which the critic or advantage normalization has to absorb.
3. So normalize. InstructGPT adds a bias so that the human demonstrations score 0 on average before RL starts. Ziegler et al. rescale the reward to mean 0 and variance 1 under the initial policy. This also makes $\beta$ mean roughly the same thing from one reward model to the next.

The scale of the reward, unlike its zero point, is fixed by the data: the sigmoid’s slope is 1, so a gap of 1 always means 73%. As the Boltzmann reading predicted, noisier labelers produce a reward model with smaller gaps.

Accuracy and calibration

Two numbers tell you whether a reward model is any good.

• Accuracy: on held-out pairs, how often is $r_\phi(x, y_w) > r_\phi(x, y_l)$? Because labels are noisy, 100% is impossible. Suppose a pair is truly preferred one way with probability $p$. Even a perfect model, which always predicts the more likely side, is right only $\max(p, 1-p)$ of the time. Averaged over pairs, that is the accuracy ceiling. It sits somewhat above the inter-labeler agreement $p^2 + (1-p)^2$: for $p = 0.75$, the ceiling is 75% but two labelers agree only 62.5% of the time, because the reward model predicts the average labeler, while each of the two labelers adds its own noise. Stiennon et al.’s 6.7B reward model, tested on a new domain (news articles), agreed with labelers about as often as labelers agreed with each other. On human preference data, accuracies in the high 60s or low 70s are normal, not a sign of failure.
• Calibration: when the model says $\sigma(\Delta) = 0.8$, do labelers prefer $y_w$ about 80% of the time? Bai et al. (2022a) check this and find their preference models trained on helpfulness data very well calibrated, and slightly less so when harmlessness data is mixed in. Calibration matters because RL treats reward differences as meaningful amounts, not just orderings.

Now watch a reward model learn. The demo has six responses with hidden true qualities (rings). A simulated labeler compares random pairs with Bradley–Terry noise, and a reward model with one free score per response (dots, all starting at 2) is fit by gradient descent on the loss above, plus a tiny L2 penalty on the spread of the scores. κ (kappa) sets how consistent the simulated labeler is: labels follow $\sigma(\kappa(s_i - s_j))$ for true qualities $s$, so κ plays the role of an inverse temperature. The “spread ratio” is the standard deviation of the fitted scores divided by that of the true ones.

Optimizing the reward with PPO

The objective, again

With a reward model in hand, step 3 solves the problem from Chapter 11:

Its exact optimum is the reference model tilted by the reward, $\pi^*(y\mid x) \propto \pi_\text{ref}(y\mid x)\, e^{r_\phi(x, y)/\beta}$ (Chapter 10 derives it as a Gibbs variational principle; Chapter 11 restates it for language models). We can’t sample from $\pi^*$ directly: its normalizer sums over every possible response. So we use RL to move $\pi_\theta$ toward it, one gradient step at a time.

The KL term matters more here than anywhere else in the course. The reward model was trained on answers from $\pi_\text{SFT}$. Far from those answers it has never seen a label, and its scores there are extrapolation. The KL leash keeps the policy where the reward model knows what it’s talking about. (Chapter 11 gives two more reasons: it preserves fluency and diversity, and it stops the policy from collapsing onto one answer.) The lab shows what happens when you cut the leash.

Four models

PPO-based RLHF juggles four networks of similar size:

There is also $\pi_\text{old}$ from Chapter 9, the policy that generated the current batch, but it isn’t a fifth network: it is just the policy’s log-probabilities, recorded once at sampling time. Two of the four models need gradients and optimizer states, which is why PPO-RLHF is memory-hungry (Exercise 12.8, and Chapter 20 for systems).

Stiennon et al. give the critic its own parameters rather than a value head on the policy: when they shared them, early value-function updates damaged the pretrained policy.

The language model as an RL agent

From Chapter 11: the state $s_t$ is the prompt plus the tokens so far, $(x, y_{<t})$; the action is the next token $y_t$; transitions are deterministic (append the token); the episode ends at the end-of-sequence token or a length limit. Episodes are short and finite, so there is no discounting: $\gamma = 1$. InstructGPT states “no discount is applied”.

Per-token rewards

The reward model gives one number for the whole response. The KL penalty, on the other hand, splits naturally over tokens, because the log-probability of a sequence is a sum of token log-probabilities:

So PPO-RLHF gives every token $t$ of a response with $T$ tokens a per-token reward

The rewards of a response add up to $r_\phi(x, y) - \beta \log\frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}$. Its expectation over $y \sim \pi_\theta$ is exactly the objective above, since the expected log-ratio is the KL. In Chapter 11’s terms, each token’s penalty is the $k_1$ estimator of the KL, and the policy pays it as it goes.

Worked example. Prompt: “Capital of France?”. The policy samples a three-token response, “Paris”, ”.”, EOS. The reward model scores it $r_\phi = 1.5$, and $\beta = 0.1$.

The total is $1.47 = 1.5 - 0.1 \times 0.3$. The policy has made “Paris” more likely than the reference does, so it pays 0.05 for that token. It made ”.” less likely, so that token earns a small refund: single-token log-ratios can be negative, even though their expectation, the KL, can’t.

Two details. These rewards are computed once, at sampling time, from $\pi_\text{old}$‘s log-probabilities, with no gradient flowing through them: the KL is part of the reward, not the loss. (GRPO, in Chapter 15, instead adds a KL estimate to the loss, which gives a different gradient; Chapter 16 compares the two.) And the reward model must actually see an end-of-sequence token, because that’s where it reads its score: implementations give a fixed penalty to responses that hit the length limit without one (Huang et al. 2024 call this the “EOS trick”).

Advantages from a value head

With per-token rewards and a critic, advantages are computed exactly as in Chapter 8: TD errors $\delta_t = r_t + V(s_{t+1}) - V(s_t)$ with $\gamma = 1$, combined by GAE, $\hat A_t = \sum_{l \ge 0} \lambda^l \delta_{t+l}$. The state after the last token is terminal, with value 0.

What does the critic predict? At each prefix, the total reward still to come: the final reward-model score, minus the KL penalties still to be paid. In plain words, “how good will this half-written answer end up?” Initializing it from the reward model gives it a head start, since the reward model already judges text.

Worked example, continued. Say the critic outputs $V(s_1) = 1.0$ before “Paris”, $V(s_2) = 1.2$ before ”.”, and $V(s_3) = 1.4$ before EOS. Then

With $\lambda = 1$ the advantages are Monte Carlo returns minus values: $\hat A_3 = 0.10$, $\hat A_2 = 0.32$, $\hat A_1 = 0.47$ (check: the whole return 1.47 minus $V(s_1) = 1.0$). With $\lambda = 0.95$: $\hat A_3 = 0.10$, $\hat A_2 = 0.22 + 0.95 \times 0.10 = 0.315$, $\hat A_1 = 0.15 + 0.95 \times 0.315 \approx 0.449$. All three tokens get pushed up, “Paris” the most: the critic didn’t expect this answer to go so well.

The PPO-RLHF loop

Putting it together:

PPO-based RLHF  (InstructGPT-style)
inputs: π_SFT, reward model r_φ, prompt set D, KL coefficient β, clip ε, GAE λ
π_θ ← π_SFT;   π_ref ← π_SFT (frozen);   V_ψ ← copy of r_φ (trainable)
for iteration = 1, 2, ...:
  1. Rollout: sample prompts x ~ D; generate y ~ π_θ(·|x) (temperature 1, max length L).
     Record per token:  logp_old_t = log π_θ(y_t | x, y_<t),   V_old_t = V_ψ(x, y_<t)
  2. Score:   s = r_φ(x, y) at the EOS token (fixed penalty if there is no EOS)
              ref_t = log π_ref(y_t | x, y_<t)
  3. Rewards: r_t = −β·(logp_old_t − ref_t);   r_T += s
  4. Advantages: Â_t = GAE(γ = 1, λ) from r_t and V_old_t;   targets R_t = Â_t + V_old_t
                 normalize  over the batch's tokens
  5. Update, for n_epochs passes over minibatches:
       ρ_t      = exp(log π_θ(y_t|·) − logp_old_t)
       L_policy = − mean over tokens of min(ρ_t Â_t, clip(ρ_t, 1−ε, 1+ε) Â_t)
       L_value  =   mean over tokens of (V_ψ(x, y_<t) − R_t)²
       (PPO-ptx) add − γ_ptx · mean log π_θ(pretraining text)
       one optimizer step on L_policy + c·L_value (+ the ptx term)    (c ≈ 0.5: value-loss weight)
  6. Optionally adapt β toward a target KL.

Steps 4 and 5 are Chapter 9’s PPO unchanged, with tokens as time steps. “Mean over tokens” means padding and prompt tokens are masked out: only generated tokens are actions.

The details that make it work

PPO on language models has a reputation for being fiddly, and much of the craft lives in details. The most important ones:

• Small, careful updates. InstructGPT used batches of 512 rollouts split into 8 minibatches of 64, a single PPO epoch per batch, clip $\epsilon = 0.2$, sampling temperature 1, $\beta = 0.02$, and an exponential moving average of the policy weights, for 256k episodes in total.
• Reward normalization or whitening. Rescale reward-model scores (or per-batch rewards) to a standard range, so $\beta$, the learning rate and the clip mean the same thing across reward models. Advantage normalization (Chapter 9) is standard too. Xu et al. (2024) name advantage normalization, large batches and a moving-average reference model as the key factors for PPO in RLHF.
• An adaptive KL coefficient. Runs with the same fixed $\beta$ can end up at very different KLs. Ziegler et al. adapt it toward a target with a proportional controller: $e_t = \operatorname{clip}\big((\mathrm{KL}_t - \mathrm{KL}_\text{target})/\mathrm{KL}_\text{target}, -0.2, 0.2\big)$, $\beta_{t+1} = \beta_t(1 + 0.1\, e_t)$. It plays the same role as PPO-penalty’s adaptive $\beta$ in Chapter 9, but for the leash rather than the trust region.
• Length limits and the EOS trick. Cap the generation length; penalize responses that hit the cap without finishing, since the reward model can only score finished ones.
• Dropout off. With dropout, recomputing a token’s log-probability gives a different number each time, so the ratio $\rho_t$ and the KL penalty are not well defined. Ziegler et al. and later reproductions switch it off.
• PPO-ptx, against the alignment tax. InstructGPT’s RLHF models got worse than GPT-3 on some public benchmarks (SQuAD, DROP, HellaSwag, WMT French-to-English translation). The authors call this an alignment tax: a capability cost of alignment. Their fix, PPO-ptx, mixes ordinary next-token training on pretraining data into the RL updates, $\gamma_\text{ptx}\,\mathbb E_{x\sim\mathcal D_\text{pretrain}}[\log\pi_\theta(x)]$ with $\gamma_\text{ptx} = 27.8$ (the paper calls it $\gamma$; it has nothing to do with the discount). It greatly reduced the regressions without lowering labelers’ preference scores. Bai et al. (2022a) report that only their smaller models pay such a tax; their 13B and 52B RLHF models did better on most zero-shot NLP evaluations than the corresponding plain language models.

When the reward model is wrong

Everything above optimizes $r_\phi$, a model of human judgment fitted to tens of thousands of comparisons. Optimize hard enough against any model and you will find its mistakes.

Stiennon et al. (2020) measured this directly. They trained policies against one reward model with a range of KL coefficients and asked humans to judge the results. Under light optimization, real human preference improved along with the reward model’s score. Pushed further, the reward model’s score kept rising while actual preference fell, until the reward model became anti-correlated with what people wanted. Bai et al. (2022a) report that during RLHF the reward rises roughly linearly in the square root of the KL from the initial policy. Gao et al. (2022) turned these observations into scaling laws for reward model overoptimization; Chapter 19 is about them and about reward hacking in general.

The lab reproduces the shape in miniature. A tiny reward model learns, from noisy rankings, that the hidden labeler likes answers ending in an even digit. It works well on the SFT model’s answers. But it adds up a weight per digit, so it has a blind spot: more even digits always score higher, even past the point where the answer stops being well formed. Run PPO with $\beta = 0$ and, after about 30 iterations of genuine improvement, the policy discovers the blind spot. It stops emitting EOS, pads its answers with even digits (4444), and drives the reward model’s score up while the hidden labeler’s score falls to its minimum. With $\beta = 0.1$ the hack never happens, and with PPO-ptx the arithmetic skill that the labeler never asked about survives too.

The mitigations are the subject of later chapters. Keep $\beta$ honest, and watch the KL. Collect fresh comparisons on the current policy’s outputs and retrain the reward model, as Bai et al. did every week (“iterated online RLHF”). Replace human labels with AI feedback or rubrics where possible (Chapter 14). Fold the reward model into the policy and skip RL altogether (Chapter 13). Or move to tasks where the reward can be checked instead of predicted (Chapter 15).

Summary

• RLHF has three steps: SFT on demonstrations, a reward model fitted to human comparisons, and PPO against that reward model with a KL penalty to the SFT model.
• People compare more reliably than they score. A ranking of $K$ answers gives $\binom{K}{2}$ correlated pairs. Labelers agree only about 73% of the time, so labels are noisy samples.
• Bradley–Terry: $P(y_1 \succ y_2) = \sigma(r_1 - r_2)$. It follows from Gumbel noise on utilities, it is Elo with a different unit, and it is a two-state Boltzmann distribution with energy $-r$. Rewards are defined only up to a constant per prompt; their scale reflects labeler consistency.
• The reward model is an LM with a scalar head read at the last token, trained with $-\log\sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$. Each pair is weighted by how wrong the model still is. Good accuracy is near the label-noise ceiling, not near 100%; calibration matters too.
• PPO-RLHF uses four models (policy, frozen reference, frozen reward model, critic). Each token gets $-\beta\log\frac{\pi_\theta}{\pi_\text{ref}}$; the last token also gets $r_\phi(x, y)$. GAE with $\gamma = 1$ turns this into per-token advantages, and Chapter 9’s clipped loss does the rest.
• The clip is a trust region to $\pi_\text{old}$; the $\beta$ term is a leash to $\pi_\text{ref}$. Practical keys: normalized rewards and advantages, small conservative updates, an adaptive $\beta$, the EOS trick, dropout off, and PPO-ptx against the alignment tax.
• A reward model is a proxy. Optimized hard, its score keeps rising while true quality falls. The KL leash is the first line of defense.

Exercises

cd 12-rlhf
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Ouyang et al. (2022), Training language models to follow instructions with human feedback. InstructGPT: the canonical three-step recipe. Section 3 is the method; Appendix C has the hyperparameters quoted in this chapter.
• Stiennon et al. (2020), Learning to summarize from human feedback. The paper where RLHF on language models clearly worked. Read it for the careful data collection and the reward-overoptimization figure.
• Huang et al. (2024), The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. An open reproduction of Stiennon et al. that lists every small detail (EOS trick, dropout, padding, whitening). The best guide if you implement PPO-RLHF yourself.
• Ziegler et al. (2019), Fine-Tuning Language Models from Human Preferences. Short and clear. The source of the KL-in-the-reward formulation and the adaptive KL controller.
• Bai et al. (2022a), Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Anthropic’s version: preference-model calibration, iterated online RLHF, the reward-vs-√KL relation, and the helpful–harmless tension.
• Christiano et al. (2017), Deep reinforcement learning from human preferences. Where learning a reward from comparisons began, on Atari and robot tasks.
• Nathan Lambert, Reinforcement Learning from Human Feedback (online book, regularly updated). A free book on the whole post-training pipeline; its reward modeling and policy-gradient chapters complement this one.
• Bradley & Terry (1952), Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. The original model, from statistics, long before language models.

Next: Chapter 13 shows that the reward model and PPO can be folded into a single supervised loss, using the closed-form optimum $\pi^*$ and the fact that Bradley–Terry only ever sees reward differences.