Direct preference optimization

Chapter 12 built the RLHF pipeline. You collected preference pairs, fit a Bradley–Terry reward model, and ran PPO against it, with four networks in memory (policy, reference, reward model, value function), a sampling loop, per-token KL penalties, GAE, and a long list of implementation details. It works, but it’s a lot of machinery for a simple goal: make the model produce more of what people prefer.

In 2023, Rafailov and colleagues pointed out that most of the machinery is unnecessary, at least on paper. The RL step in RLHF has a known answer, a closed-form formula you met in Chapter 10 and Chapter 11. If you know the answer’s form, you can turn the problem around: instead of learning a reward and then searching for the policy that maximizes it, write the reward in terms of the policy, and fit the policy to the preferences directly. The result is direct preference optimization (DPO): one loss, computed from four log-probabilities per preference pair, trained like supervised learning. No reward model, no sampling, no PPO. The paper’s subtitle says what is going on: your language model is secretly a reward model.

This chapter derives DPO slowly, with numbers you can check, then looks hard at what it gives up. DPO is offline, so it never sees its own mistakes. It can push the probability of the preferred answers down. With few labels it ignores its own leash. Then come the variants people built to fix these problems (IPO, KTO, ORPO, SimPO, online DPO), and the contested question of whether DPO or PPO works better.

Two facts you already have

DPO is built from two results from earlier chapters. Here they are side by side.

Fact 1: the KL-regularized optimum (Chapter 10 derived it, Chapter 11 applied it to language models). For a prompt $x$, the RLHF objective is

$$\max_{\pi}\; \mathbb{E}_{y \sim \pi(\cdot\mid x)}\big[r(x, y)\big] \;-\; \beta\, \mathrm{KL}\big(\pi(\cdot\mid x) \,\Vert\, \pi_\text{ref}(\cdot\mid x)\big),$$

and its maximizer is the reference policy tilted by the exponentiated reward:

$$\pi^*(y\mid x) \;=\; \frac{1}{Z(x)}\, \pi_\text{ref}(y\mid x)\, e^{r(x,y)/\beta}, \qquad Z(x) = \sum_{y} \pi_\text{ref}(y\mid x)\, e^{r(x,y)/\beta}.$$

$Z(x)$ is a partition function: a sum over every possible response, astronomically many for a language model. Nobody can compute it. That’s why RLHF doesn’t just write down $\pi^*$ and stop; it runs PPO to approximate it by sampling.

Fact 2: the Bradley–Terry model (Chapter 12). The probability that an annotator prefers $y_w$ to $y_l$ for prompt $x$ is

$$P(y_w \succ y_l \mid x) \;=\; \sigma\big(r(x, y_w) - r(x, y_l)\big),$$

with $\sigma$ the logistic sigmoid. A reward model is fit by maximizing this likelihood on the preference data, which is the loss $-\log\sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$. Notice that only reward differences between two responses to the same prompt enter.

RLHF uses these facts in sequence: fit $r$ with Fact 2, then chase the $\pi^*$ of Fact 1 with RL. DPO’s idea is to use Fact 1 backwards.

The trick, with numbers

Take Chapter 10’s worked example and pretend it’s a language model with a single prompt and only three possible responses A, B, C. The reference policy is $\pi_\text{ref} = (0.5, 0.3, 0.2)$, the rewards are $r = (0, 1, 2)$, and $\beta = 1$. Chapter 10 computed the tilted optimum: the weights $\pi_\text{ref}\,e^{r}$ are $(0.500, 0.816, 1.478)$, so $Z = 2.793$, $\log Z = 1.027$, and $\pi^* = (0.179, 0.292, 0.529)$.

Now go the other way. Suppose someone hands you $\pi^*$ and $\pi_\text{ref}$ but not the reward. Compute $\beta \log\frac{\pi^*}{\pi_\text{ref}}$ for each response:

Response	$\pi_\text{ref}$	$\pi^*$	$\beta\log\dfrac{\pi^*}{\pi_\text{ref}}$	true $r$	$r - \beta\log Z$
A	0.5	0.179	$\ln(0.358) = -1.027$	0	$-1.027$
B	0.3	0.292	$\ln(0.973) = -0.027$	1	$-0.027$
C	0.2	0.529	$\ln(2.645) = 0.973$	2	$0.973$

The log-ratio column is the reward minus one constant, $\beta \log Z = 1.027$, the same for every response. So you don’t recover the reward exactly, but you recover every reward difference exactly: B − A $= 1$, C − A $= 2$, C − B $= 1$.

And reward differences are all Bradley–Terry needs. The probability an annotator prefers C to A is $\sigma(2 - 0) = 0.881$, and you get the same number from the policy alone: $\sigma(0.973 - (-1.027)) = \sigma(2) = 0.881$. The unknowable $Z$ never came into it, because it cancelled in the subtraction.

So here is DPO in one sentence. If the policy is the tilted optimum for some reward, that reward is $\beta \log\frac{\pi}{\pi_\text{ref}}$ (up to a per-prompt constant), so fit the policy so that this implied reward explains the preference data under Bradley–Terry.

Deriving the DPO loss

Now the general version, in four steps.

Step 1: invert the optimum. Take the log of Fact 1 and solve for the reward:

$$r(x, y) \;=\; \beta \log \frac{\pi^*(y\mid x)}{\pi_\text{ref}(y\mid x)} \;+\; \beta \log Z(x).$$

Step 2: substitute into Bradley–Terry. For two responses to the same prompt, the $\beta \log Z(x)$ terms are identical and cancel:

$$P(y_w \succ y_l \mid x) \;=\; \sigma\Big(\beta \log \frac{\pi^*(y_w\mid x)}{\pi_\text{ref}(y_w\mid x)} \;-\; \beta \log \frac{\pi^*(y_l\mid x)}{\pi_\text{ref}(y_l\mid x)}\Big).$$

The preference probability is now written entirely in terms of the optimal policy and the reference. No reward function appears.

Step 3: make the policy the thing you learn. Replace $\pi^*$ by the trainable model $\pi_\theta$. Each policy now defines a reward model, its implicit reward

$$\hat r_\theta(x, y) \;=\; \beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)},$$

and $\pi_\theta$ is exactly the KL-regularized optimum for $\hat r_\theta$ (plug $\hat r_\theta$ into Fact 1 and you get $Z = 1$ and $\pi^* = \pi_\theta$).

Step 4: maximum likelihood. Fit $\hat r_\theta$ to the preference data with the ordinary Bradley–Terry loss, as you would fit a reward model in Chapter 12. That’s the DPO loss:

$$\mathcal{L}_\text{DPO}(\theta) \;=\; -\,\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\Big[\log \sigma\Big(\beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_\text{ref}(y_w\mid x)} \;-\; \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_\text{ref}(y_l\mid x)}\Big)\Big].$$

Each term is a sequence log-probability, and (Chapter 11) a sequence log-probability is a sum of token log-probs: $\log \pi_\theta(y\mid x) = \sum_t \log \pi_\theta(y_t \mid x, y_{\lt t})$. So for each pair you need exactly four numbers: $\log\pi_\theta(y_w\mid x)$, $\log\pi_\theta(y_l\mid x)$, $\log\pi_\text{ref}(y_w\mid x)$, $\log\pi_\text{ref}(y_l\mid x)$. That takes one forward pass through each model for each of the two responses. The reference numbers don’t change during training, so you can compute them once in advance.

∂ The fine print: is fitting the policy really the same as fitting a reward and then optimizing it?

Step 3 quietly swapped “learn a reward, then find its optimal policy” for “learn a policy”. Here is why nothing is lost, in an idealized setting where both $r$ and $\pi$ can be any function.

Rewards come in equivalence classes. Call two rewards equivalent if they differ by a function of the prompt only: $r'(x, y) = r(x, y) + f(x)$. Equivalent rewards

• give the same Bradley–Terry probabilities, because $f(x)$ cancels in $r'(x, y_w) - r'(x, y_l)$ (this is the shift invariance of Chapter 12), and
• give the same optimal policy, because $e^{r'/\beta} = e^{f(x)/\beta} e^{r/\beta}$ and the factor $e^{f(x)/\beta}$ cancels when you normalize over $y$.

(The DPO paper states these two facts as its Lemmas 1 and 2.)

Each class corresponds to exactly one policy, and vice versa. Going from rewards to policies, each class maps to one optimal policy $\pi_r \propto \pi_\text{ref} e^{r/\beta}$, by the second bullet. Going back, take any policy $\pi$ that is positive wherever $\pi_\text{ref}$ is. The reward $r_\pi = \beta \log\frac{\pi}{\pi_\text{ref}}$ has $Z(x) = \sum_y \pi_\text{ref}\,\frac{\pi}{\pi_\text{ref}} = 1$, so its optimal policy is $\pi$ itself. And any reward whose optimal policy is $\pi$ satisfies $r = \beta\log\frac{\pi}{\pi_\text{ref}} + \beta\log Z(x)$, so it is in the class of $r_\pi$. Classes and policies are in one-to-one correspondence.

So the two problems are the same problem. RLHF solves $\max_r \text{(BT log-likelihood of } r)$ and then outputs $\pi_r$. DPO solves $\max_\pi \text{(BT log-likelihood of } r_\pi)$. Because the map between classes and policies is a bijection, and the likelihood depends only on the class, the DPO optimum is $\pi_{\hat r}$, where $\hat r$ is the maximum-likelihood reward. That is exactly what RLHF would return if its RL step found the exact optimum. (The DPO paper’s Theorem 1 states the representation result.)

Where the idealization breaks. In practice the reward model and the policy are different neural networks trained by different procedures on finite data. The Bradley–Terry maximum-likelihood solution is not unique on responses the data never mentions, and the two pipelines fill in that gap differently: PPO’s KL penalty is measured on responses the policy actually samples, while DPO’s loss says nothing at all about responses outside the pairs. “Equivalent in the limit” does not mean “equivalent in practice”. The section on what DPO gives up returns to this.

Why the pair must share a prompt

The cancellation in Step 2 is the whole trick, and it needs both responses to be answers to the same $x$. Suppose you tried to compare $y_1$ for prompt $x_1$ with $y_2$ for prompt $x_2$. Then

$$r(x_1, y_1) - r(x_2, y_2) = \beta \log \frac{\pi^*(y_1\mid x_1)}{\pi_\text{ref}(y_1\mid x_1)} - \beta \log \frac{\pi^*(y_2\mid x_2)}{\pi_\text{ref}(y_2\mid x_2)} + \beta\log\frac{Z(x_1)}{Z(x_2)},$$

and the last term, a ratio of two intractable sums, refuses to go away. DPO can only learn from comparisons within a prompt. That matches how preference data is collected anyway: annotators see one prompt and two answers.

A worked example of the loss

Take $\beta = 0.1$ and one pair with these sequence log-probs:

	$\log \pi_\theta$	$\log \pi_\text{ref}$	log-ratio	implicit reward $\hat r = \beta \times$ log-ratio
chosen $y_w$	$-12.0$	$-13.0$	$+1.0$	$+0.1$
rejected $y_l$	$-15.0$	$-14.0$	$-1.0$	$-0.1$

The policy has made the chosen response $e^{1} \approx 2.7$ times more likely than the reference does, and the rejected one $e$ times less likely. The implicit reward margin is $\hat r_w - \hat r_l = 0.2$, the implicit reward model says $P(y_w \succ y_l) = \sigma(0.2) = 0.550$, and the loss is $-\log\sigma(0.2) = \log(1 + e^{-0.2}) = 0.598$.

Two numbers to remember. At the start of training $\pi_\theta = \pi_\text{ref}$, every log-ratio is 0, every margin is 0, and the loss is $\log 2 = 0.693$ for every pair. Any other value at step 0 means a bug. And notice that the raw log-probs ($-12$, $-15$) are large and negative, as they are for any real response of a few dozen tokens, but only the differences from the reference matter.

What the gradient does

The DPO loss looks like a classification loss. To see what it does to the policy, differentiate it. Write $u = \hat r_\theta(x, y_w) - \hat r_\theta(x, y_l)$ for the margin. Since $\frac{d}{du}\big[-\log\sigma(u)\big] = -(1 - \sigma(u)) = -\sigma(-u)$ and $\nabla_\theta u = \beta\big(\nabla_\theta \log\pi_\theta(y_w\mid x) - \nabla_\theta \log\pi_\theta(y_l\mid x)\big)$ (the reference terms are constants),

$$\nabla_\theta \mathcal{L}_\text{DPO} \;=\; -\,\beta\; \mathbb{E}\Big[\underbrace{\sigma\big(\hat r_\theta(x, y_l) - \hat r_\theta(x, y_w)\big)}_{\text{weight: how wrong is the implicit RM?}} \Big(\underbrace{\nabla_\theta \log \pi_\theta(y_w\mid x)}_{\text{raise the winner}} \;-\; \underbrace{\nabla_\theta \log \pi_\theta(y_l\mid x)}_{\text{lower the loser}}\Big)\Big].$$

Gradient descent on the loss moves $\theta$ along the negative of this, so each pair does three things:

1. Raise the winner. $+\nabla\log\pi_\theta(y_w\mid x)$ is the supervised fine-tuning gradient on $y_w$.
2. Lower the loser. $-\nabla\log\pi_\theta(y_l\mid x)$ pushes the rejected response down. This “negative gradient” is something SFT can’t do: SFT only ever learns from good examples.
3. Weight by surprise. The factor $\sigma(\hat r_l - \hat r_w)$ is the probability the implicit reward model assigns to the wrong ordering. Pairs it already ranks confidently get almost no weight; pairs it gets wrong get almost full weight.

Here is the weight as the margin grows:

implicit margin $u = \hat r_w - \hat r_l$	$-2$	$-1$	$0$	$1$	$2$	$4$
loss $-\log\sigma(u)$	2.127	1.313	0.693	0.313	0.127	0.018
weight $\sigma(-u)$	0.881	0.731	0.500	0.269	0.119	0.018

The weight is what keeps DPO from running away. Without it, “push the winner up and the loser down with equal force forever” has no stopping point. The DPO paper reports that a naive version without the weighting coefficient can make the language model degenerate. With it, each pair contributes less and less once the implicit reward model is confident about it. (Less and less, but never zero: with labels that never disagree, the push fades without stopping, which is the problem the section on deterministic preferences takes up.)

The algorithm in practice

DPO
input: preference pairs D = {(x, y_w, y_l)}, SFT model π_SFT, β (e.g. 0.1)
1. π_ref ← copy of π_SFT, frozen;   π_θ ← π_SFT, trainable
2. (optional) precompute log π_ref(y_w|x) and log π_ref(y_l|x) for every pair
3. for each minibatch of pairs:
4.     lw ← Σ_t log π_θ(y_w,t | x, y_w,<t)          # masked sum over response tokens
5.     ll ← Σ_t log π_θ(y_l,t | x, y_l,<t)
6.     u  ← β·[(lw − log π_ref(y_w|x)) − (ll − log π_ref(y_l|x))]
7.     loss ← mean(−log σ(u))                         # use a stable log-sigmoid
8.     gradient step on loss
9.     log: reward accuracy mean(u > 0), mean margin, mean lw and ll

Some practical points, all of which you will meet in the lab:

• The reference is the SFT model. DPO assumes $\pi_\text{ref}$ is a reasonable model of the distribution the preference data came from. The DPO paper recommends, when no SFT model is available, first fine-tuning on the preferred responses to get one.
• Masking. Prompt tokens and padding are excluded from the sums, exactly as in Chapter 11. Llama 3’s DPO also masked out special formatting tokens (headers and end-of-turn tokens) that appear in both responses.
• Hyperparameters. $\beta$ around 0.1 is the common default; Llama 3 reports $\beta = 0.1$ with learning rate $10^{-5}$. Learning rates are small and training is short, usually one to a few epochs over the pairs, because DPO overfits (more below).
• What to watch. The reward accuracy (fraction of pairs with $\hat r_w > \hat r_l$) and the margin tell you whether the implicit reward model is learning. The chosen and rejected log-probs, $\log\pi_\theta(y_w\mid x)$ and $\log\pi_\theta(y_l\mid x)$, tell you how it is learning. You would hope the chosen one goes up. Often it doesn’t.

Here is how this compares with the pipeline of Chapter 12:

	RLHF with PPO (Chapter 12)	DPO
Stages after SFT	train a reward model, then run RL	one training run
Networks in memory	policy, reference, reward model, value function	policy and reference (reference log-probs can be precomputed)
Generation during training	every step: sample responses, score them	none
Data	preference pairs (for the RM) plus prompts (for RL)	preference pairs only
Where the KL leash acts	explicit penalty, on responses the policy samples	built into the loss through the derivation
Learns from its own outputs?	yes: on-policy	no: offline
Typical failure	the policy exploits reward-model errors	overfitting the pairs, likelihood displacement, drift on unseen responses

DPO is cheaper and simpler. The last two rows are the price.

Watch DPO train

The demo below is a whole “language model” for a single prompt with a handful of possible responses, small enough to see everything. Each scenario fixes $\pi_\text{ref}$, a hidden true reward $r$ (used only to draw the RLHF target $\pi^*$, the thick horizontal ticks on the bars), and a preference dataset. A pair that annotators label "$y_i \succ y_j$" with probability $p$ contributes its average loss, $p$ times the loss with $y_i$ as winner plus $1-p$ times the loss with $y_j$ as winner. The demo minimizes this with Adam on all pairs at once (no sampling), so the curves are smooth and repeatable. The readout below the table shows the step, the loss, the mean implicit-reward margin on the pairs, and two KL divergences: how far $\pi_\theta$ has moved from $\pi_\text{ref}$, and how far it still is from $\pi^*$.

In scenarios 1 and 2 the policy is tabular: one free logit per response. In scenario 3 the four responses are described by 2-dimensional feature vectors and the policy is $\pi_\theta(y) \propto \pi_\text{ref}(y)\,e^{\phi(y)\cdot\theta}$. That is a cartoon of a real network, where similar responses have similar internal representations and therefore move together.

Data1 · Every pair, Bradley–Terry labels2 · Two pairs, always the same label3 · Similar responses ("No." ≻ "Never.")LossDPOIPODPO + 0.5 × NLL on the chosenβ =

1.00

Train ▶+100 stepsReset

π_ref  π_θ (trained)  π* (RLHF optimum)

solid: log π_θ(y)   dashed: log π*(y)

response	in the data	true r	πref	πθ	π*	implicit reward β·log(πθ/πref)

What to try: (1) Scenario 1, DPO, β = 1: press Train. π_θ converges onto the π* ticks (the thick horizontal lines), (0.179, 0.292, 0.529) as in the worked example, the implicit rewards become (−1.027, −0.027, 0.973), and KL(π_θ ‖ π*) goes to 0. Change β to 0.3 and train again: π* changes, but the implicit rewards still differ by exactly 1 and 2, the true reward gaps. (2) Scenario 2: response B is the winner of one pair, yet its log-probability falls (the curve labeled B). D, the best response, appears in no pair and is never learned. The margin grows without limit, and the final policy is the same for β = 0.1 and β = 1. Switch the loss to IPO: the margin stops at 0.5. (3) Scenario 3: the data says only 'No.' ≻ 'Never.', yet 'Yes.' takes over and 'No.' first rises, then collapses. Switch to DPO + NLL and train again.

Scenario 1 confirms the derivation: when every pair is labeled according to Bradley–Terry with the true reward (think: infinitely many annotators per pair), DPO’s minimizer is exactly the RLHF optimum $\pi^*$, without ever seeing the reward or $Z$. Scenarios 2 and 3 are what happens with realistic data, and they lead straight to the next section.

What DPO gives up

It’s offline

DPO trains on a fixed dataset of pairs that some other policy generated (often a mixture of older models, or the SFT model before training). It never samples from $\pi_\theta$, so it never finds out what $\pi_\theta$ actually does. This is Chapter 10’s offline RL problem in a new form.

Look at what the loss constrains. It involves $\pi_\theta$ only through the log-probabilities of the responses in the pairs. Every other response, which is nearly all of them, is unconstrained, except that probabilities must sum to one. If raising $y_w$ relative to $y_l$ happens to also raise some response $y'$ that appears nowhere in the data, the loss doesn’t notice, and nothing in DPO will ever sample $y'$ to find out it’s bad.

Xu et al. (2024) make this precise. They prove that every policy PPO-based RLHF can produce also minimizes the DPO loss, but not the reverse: DPO’s set of solutions is strictly larger. Their toy example has three responses with $\pi_\text{ref} = (0.5, 0.5, 0)$ and one preference, $y_1 \succ y_2$. The policy $(0.1, 0, 0.9)$ minimizes the DPO loss (the loss only needs $\pi(y_2) = 0$) while putting 90% of its mass on $y_3$, a response the reference never produces and no annotator ever saw. A KL-penalized RL policy can’t do this, because $y_3$ would have infinite KL from the reference. Scenario 2 in the demo shows the milder version: the best response D gets nothing, because nobody ever compared it with anything.

PPO-based RLHF has the opposite profile. It samples from the current policy, so it finds out what the policy does, and the reward model judges those very samples. Its weakness is that the reward model may be wrong about them (reward hacking, Chapter 19). DPO trades “a wrong judge of new samples” for “no judge of new samples”.

Deterministic preferences: the leash goes slack

In the derivation, $\beta$ is the strength of the KL leash: large $\beta$ keeps $\pi_\theta$ near $\pi_\text{ref}$. But look at what the loss does with a pair whose label is always the same. If a fraction $p$ of annotators prefer $y_w$, the expected loss on that pair, as a function of the margin $u$, is $-p\log\sigma(u) - (1-p)\log\sigma(-u)$. It is minimized where $\sigma(u) = p$, at

$$u^* = \log\frac{p}{1-p}.$$

(The second term is there because a fraction $1-p$ of the time the label is flipped, and then the margin enters the loss with the opposite sign.) With $p = 0.9$ that’s a finite margin, $u^* = \ln 9 = 2.2$. With $p = 1$ there is no minimum: the loss keeps decreasing as $u \to \infty$. Since $\pi_\theta(y_w\mid x) \le 1$, the chosen log-ratio can never exceed $\log\frac{1}{\pi_\text{ref}(y_w\mid x)}$, so the only way to make $u$ infinite is $\pi_\theta(y_l\mid x) \to 0$: the rejected response is driven out completely. Whatever $\beta$ is, the optimizer drives the log-ratios to infinity. $\beta$ only sets the speed, not the destination, and the KL regularization that the derivation promised has no effect.

And $p = 1$ is the normal case, not an edge case. In a real dataset each pair is labeled once, so the empirical preference probability of every pair is exactly 1, even when the true probability is 0.6. Azar et al. (2023) used this argument to motivate IPO (identity preference optimization), which replaces the log-sigmoid with a squared loss that has a finite target. Scenario 2 in the demo shows both behaviors: DPO’s margin grows without limit, while IPO’s stops at a fixed value.

In practice the damage is limited by early stopping: DPO is trained for one or a few epochs with a small learning rate, so it never gets close to the infinite optimum. That works, but it means the real regularizer is the training schedule, not $\beta$.

Likelihood displacement: the chosen response can go down

The DPO loss depends on the difference of two log-ratios. Nothing in it says $\log\pi_\theta(y_w\mid x)$ has to go up. If pushing $y_l$ down is “cheaper” than pushing $y_w$ up, the optimizer can widen the margin while both log-probabilities fall, as long as $y_l$ falls faster. The probability mass has to go somewhere, and it goes to responses outside the pair.

This is observed in real training runs, not only in toys. Pal et al. (2024) reported that DPO can lower the likelihood of preferred responses, especially when the two responses in a pair differ by only a few tokens. Razin et al. (2025) studied it under the name likelihood displacement. Their examples are striking. Training a model to prefer “No” over “Never” sharply raised the probability of “Yes”. In a safety setting, DPO on refusal data lowered Llama-3-8B-Instruct’s refusal rate from 74.4% to 33.4%. They trace the effect to pairs whose two responses have very similar hidden representations.

Why similarity matters is easy to see in the demo’s feature model. With $\pi_\theta(y) \propto \pi_\text{ref}(y)e^{\phi(y)\cdot\theta}$ and one pair, the gradient of the margin is

$$\nabla_\theta\big[\log\pi_\theta(y_w) - \log\pi_\theta(y_l)\big] = \phi(y_w) - \phi(y_l),$$

because the normalizer’s gradient cancels in the difference. So $\theta$ moves along $d = \phi(y_w) - \phi(y_l)$. When $y_w$ and $y_l$ are nearly identical (“No.” and “Never.”), $d$ is a short vector that encodes only the small way they differ, and it can point anywhere, including toward a response with the opposite meaning. Each response’s logit changes by $\phi(y)\cdot d$ per unit step. If “Yes.” has a larger $\phi\cdot d$ than “No.” does, “Yes.” grows fastest and eventually takes mass from everything, including the “chosen” “No.”. That’s scenario 3. A real network’s representations aren’t 2-D, but the mechanism (the update direction is a difference of two similar representations) is the one Razin et al. describe.

You’ll see the same thing in the lab with a real (tiny) transformer. On correct-versus-wrong arithmetic answers, plain DPO with $\beta = 0.1$ separates chosen from rejected on more than 90% of training pairs, while the held-out log-probability of the correct answers falls from $-0.67$ to about $-2.5$ and the model gets fewer problems right than before training. The fixes are simple:

• Add a likelihood term on the chosen response. $\mathcal{L} = \mathcal{L}_\text{DPO} - \alpha\,\frac{1}{\lvert y_w\rvert}\log\pi_\theta(y_w\mid x)$. Llama 3 used this with $\alpha = 0.2$. Pang et al. (2024), training reasoning with iterated DPO on correct versus incorrect chains of thought, found the NLL term crucial.
• Penalize any drop of the chosen log-ratio below zero (DPOP, the “DPO-Positive” loss of Pal et al.).
• Clean the data: Razin et al. filter out pairs whose responses are too similar.
• Sample from the policy (online DPO, below), so the mass that leaks to unexpected responses gets noticed and pushed back.

Length and other shortcuts

Because the implicit reward is a sum of per-token log-ratios, DPO can be influenced by response length. Preference datasets often favor longer answers (annotators and LLM judges have a verbosity bias, Chapter 14), and Park et al. (2024) found that DPO strongly exploits this, producing longer responses without more quality. They tie it to the same out-of-distribution problem: DPO extrapolates the “longer is preferred” pattern beyond the data. Their fix adds a length regularizer. Length-normalized variants (SimPO, and the length-normalized DPO of Tülu 3, below) attack the same issue.

The variant family

DPO started a flood of “direct alignment” losses. They differ in four choices: what data they need (pairs, or single responses labeled good or bad), whether they need a reference model, how they normalize for length, and what shape of loss they put on the margin. Here are the ones you will see named in papers, in our notation, with $h_\theta(x, y) = \log\frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}$ for the raw log-ratio.

Method	Data	Reference?	Loss, per example	Main idea
DPO (Rafailov et al. 2023)	pairs	yes	$-\log\sigma\big(\beta h_\theta(x,y_w) - \beta h_\theta(x,y_l)\big)$	Bradley–Terry on the implicit reward
IPO (Azar et al. 2023)	pairs	yes	$\big(h_\theta(x,y_w) - h_\theta(x,y_l) - \tfrac{1}{2\beta}\big)^2$	finite target margin, so deterministic labels can’t push it to infinity
KTO (Ethayarajh et al. 2024)	single responses labeled desirable / undesirable	yes	$\lambda_y - v(x,y)$, see below	no pairs needed; loss shaped by prospect theory
ORPO (Hong et al. 2024)	pairs	no	$-\log p_\theta(y_w) - \lambda\log\sigma\Big(\log\frac{\text{odds}_\theta(y_w)}{\text{odds}_\theta(y_l)}\Big)$	SFT and preference learning in one stage
SimPO (Meng et al. 2024)	pairs	no	$-\log\sigma\Big(\frac{\beta}{\lvert y_w\rvert}\log\pi_\theta(y_w\mid x) - \frac{\beta}{\lvert y_l\rvert}\log\pi_\theta(y_l\mid x) - m_0\Big)$	length-normalized reward, target margin $m_0$
DPO + NLL (Pang et al. 2024; used in Llama 3)	pairs	yes	$\mathcal{L}_\text{DPO} - \alpha\,\frac{1}{\lvert y_w\rvert}\log\pi_\theta(y_w\mid x)$	keep the chosen response likely
Online / iterative DPO (e.g. Guo et al. 2024)	pairs sampled from the current policy	yes	DPO	fresh on-policy data each round

A few words on each.

IPO. Azar et al. show that DPO and RLHF both rest on the Bradley–Terry assumption, which turns a preference probability $p$ into a reward difference through the logit function $\log\frac{p}{1-p}$. That function is infinite at $p = 1$, and that’s the source of the infinite margins. IPO optimizes preference probabilities directly (their $\Psi$ is the identity) and arrives at a squared loss that regresses the log-ratio margin to $1/(2\beta)$. With label probability $p$ the expected IPO loss is minimized at margin $(2p-1)/(2\beta)$: finite even at $p = 1$. Notice that IPO’s $\beta$ (the paper writes $\tau$) is a different knob from DPO’s: the target is a margin in raw log-ratio units, and the optimum is not the RLHF $\pi^*$ (the demo’s scenario 1 with IPO shows this).

KTO. Pairs are expensive. Often you only have thumbs up or thumbs down on single responses. KTO (named for Kahneman and Tversky’s prospect theory, a model of how people value gains and losses relative to a reference point) works with that. With the implicit log-ratio $h_\theta(x,y)$ and a reference point $z_0$ (an estimate of $\mathrm{KL}(\pi_\theta \Vert \pi_\text{ref})$ computed from the batch and not backpropagated through), the “value” of a response is $v = \lambda_D\,\sigma\big(\beta(h_\theta - z_0)\big)$ if it is desirable and $v = \lambda_U\,\sigma\big(\beta(z_0 - h_\theta)\big)$ if not, and the loss is $\lambda_y - v$. A desirable response is pushed up until it beats the reference point, an undesirable one down until it is below it. The weights $\lambda_D$, $\lambda_U$ (default 1) let you correct for unbalanced numbers of good and bad examples. The paper reports matching or exceeding pair-based methods from 1B to 30B parameters.

ORPO. It removes the reference model and the separate SFT stage. Training is SFT on $y_w$ plus a penalty on the odds ratio, where $p_\theta(y)$ is the per-token geometric mean probability $\exp\big(\frac{1}{\lvert y\rvert}\sum_t\log\pi_\theta(y_t\mid x, y_{\lt t})\big)$ and $\text{odds}_\theta(y) = \frac{p_\theta(y)}{1-p_\theta(y)}$. With no $\pi_\text{ref}$ there is no explicit leash; the SFT term keeps the model anchored to the chosen responses. The paper used $\lambda$ between 0.1 and 0.25 for its main models.

SimPO. The implicit reward is the average per-token log-probability, scaled by $\beta$, with no reference model, and the loss asks the chosen to beat the rejected by a margin $m_0$ (the paper’s $\gamma$, renamed here because $\gamma$ is our discount factor). The average removes the advantage long responses get from summing more tokens. The paper suggests $\beta$ around 2–2.5 and $m_0$ between 0.5 and 1.5, finds that removing the length normalization hurts most, and reports gains over DPO on AlpacaEval 2 and Arena-Hard. Tülu 3 used a related length-normalized DPO, which keeps the reference but divides each log-ratio by the response length (with $\beta = 5$, since per-token log-ratios are small).

Which is best? Each paper reports wins on its own benchmarks, and independent comparisons disagree. In the Tülu 3 report, length-normalized DPO beat both standard DPO and SimPO on their evaluation suite. Treat the table as a menu of fixes for specific failure modes: IPO for overfitting to deterministic labels, KTO for unpaired data, ORPO and SimPO for dropping the reference model’s memory and compute, length normalization for verbosity, NLL for displacement.

Online and iterative DPO

Most of DPO’s problems come from one fact: the pairs were not sampled from the model being trained. The obvious fix is to sample them from it.

Online DPO (Guo et al. 2024, “Direct Language Model Alignment from Online AI Feedback”) samples two responses from the current policy for each prompt, asks an LLM annotator which is better, and takes a DPO step on that fresh pair. The paper reports that this beats offline DPO, IPO and SLiC (another pairwise loss, a hinge on the log-probability margin), and RLHF, in human evaluations. Iterative DPO does the same in a few large rounds instead of every step: train with DPO, sample new responses from the trained model, label them (with humans, a reward model, a judge, or a verifier), train again. Llama 3 ran several such rounds, re-collecting preferences on the latest model each time. Self-rewarding language models (Yuan et al. 2024) let the model judge its own samples between rounds. Iterative reasoning preference optimization (Pang et al. 2024) builds the pairs from correct and incorrect chains of thought checked against the known answer.

Online DPO is no longer offline RL, and it has stopped being “RL-free” in the sense that mattered: you now need a generation loop and a source of labels for new samples, which is most of the machinery DPO set out to remove. What remains different from PPO is the loss: a contrastive loss on pairs, with no value function and no explicit KL penalty.

DPO or PPO? The empirical picture

This is contested, and the honest answer is “it depends on the data, the tuning, and what you measure.” Here are the main data points, with what each one actually claims.

• Xu et al. (2024), “Is DPO Superior to PPO for LLM Alignment?” Beyond the theorem above, they find that a carefully tuned PPO surpasses the other alignment methods they test in all their settings, from dialogue to competitive programming. They identify three things that made PPO work: advantage normalization, large batches, and an exponential moving average of the reference model.
• Ivison et al. (2024), “Unpacking DPO and PPO”. A controlled comparison across many preference datasets. Their ranking of what matters: the quality of the preference data first, then the learning algorithm, then reward model quality, then extra unlabeled prompts. PPO beat DPO, by up to 2.5% in math and 1.2% in general domains, while better data was worth up to 8% on instruction following and truthfulness.
• Llama 3 (Grattafiori et al. 2024). Meta’s post-training used SFT, rejection sampling and DPO over several rounds. The report says the team explored on-policy algorithms such as PPO but found that DPO needed less compute for large models and performed better, especially on instruction-following benchmarks like IFEval.
• Tülu 3 (Lambert et al. 2024). AI2’s open recipe is SFT, then length-normalized DPO on largely on-policy preference pairs (responses from the SFT model and other models, judged by an LLM), then RL with verifiable rewards. In their controlled comparison, PPO came out comparable to DPO, slightly lower, at a much higher compute cost, so they kept DPO for preference tuning and used PPO-style RL for the verifiable-reward stage.
• On-policy data matters more than the loss. Tajwar et al. (2024) argue that methods using on-policy samples or a “negative gradient” (pushing some responses down) do better than offline maximum-likelihood objectives, because they move probability mass between responses faster. Online DPO’s results (above) point the same way.

The pattern most practitioners take from this: with a fixed offline dataset and limited compute, DPO is a strong, cheap default. With the ability to sample from the policy and score the samples, on-policy methods tend to win, and online DPO closes much of the gap.

Where things stand in late 2026. DPO remains a standard step in open recipes: Olmo 3 still runs SFT, then DPO, then RL with verifiable rewards, building its preference pairs with the “delta learning” idea of Geng et al. (2025) (pair a stronger model’s response with a weaker model’s; what matters is the quality gap). Qwen2.5 used offline DPO on pairs built from execution feedback and answer matching before its online RL stage. But for reasoning, the center of gravity moved to on-policy RL with verifiable rewards and GRPO-style algorithms (Chapter 15). In a June 2026 Interconnects conversation, Nathan Lambert and Finbarr Timbers described DPO as largely disappearing from leading frontier recipes, while noting it can still be a compute-efficient choice and that the shift may owe something to fashion rather than careful ablations. No published study settles the question.

Summary

• The trick: the KL-regularized optimum $\pi^* \propto \pi_\text{ref}e^{r/\beta}$ can be inverted, $r = \beta\log\frac{\pi^*}{\pi_\text{ref}} + \beta\log Z(x)$. In Bradley–Terry only reward differences within a prompt appear, so $Z(x)$ cancels.
• The loss: $\mathcal{L}_\text{DPO} = -\mathbb{E}\big[\log\sigma(\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l))\big]$ with the implicit reward $\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_\text{ref}}$. It needs four sequence log-probs per pair and no sampling. At initialization it equals $\log 2$.
• The gradient raises the winner, lowers the loser, and weights each pair by $\sigma(\hat r_l - \hat r_w)$, the implicit reward model’s probability of getting it wrong.
• In the ideal limit (all pairs, Bradley–Terry labels, an expressive model), DPO’s optimum is exactly the RLHF optimum. In practice it is not.
• What it gives up: it is offline (no exploration; unseen responses are unconstrained); with deterministic labels the margin is pushed to infinity and $\beta$ stops acting as a leash (IPO’s motivation); and likelihood displacement can lower the chosen response’s probability and move mass to unrelated, sometimes opposite, responses.
• Variants fix specific failures: IPO (finite target), KTO (unpaired labels), ORPO and SimPO (no reference model; SimPO length-normalizes), DPO + NLL (displacement), online/iterative DPO (on-policy data).
• Empirically it’s contested: tuned PPO often wins in controlled studies, DPO is far cheaper and has been used by Llama 3, Tülu 3 and Olmo 3, and on-policy data matters as much as the choice of loss.

Exercises

Exercise 13.1What disappears, and what doesn'tConcept●○○

(a) List everything an RLHF-with-PPO run needs during training that a DPO run does not (models in memory, computation per step, data).

(b) DPO still needs a reference model. What role does it play, and could you drop it? Name a method that does.

(c) A colleague says: “DPO is RL without the RL.” In what sense is that right, and in what important sense is DPO not reinforcement learning as Chapter 1 defined it?

Solution

(a) PPO-based RLHF needs a reward model and a value function in memory (on top of the policy and the reference), a generation step every iteration (sample responses from the current policy, then score them with the reward model), per-token rewards with a KL penalty, GAE advantages, and the PPO clipped update over several epochs. It needs prompts for the RL stage in addition to the preference data used to train the reward model. DPO needs only the preference pairs, the policy, and the reference, and the reference’s log-probs can be computed once before training.

(b) The reference anchors the implicit reward: $\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_\text{ref}}$ measures how much the policy has changed a response’s probability, and the derivation’s KL leash is a leash to $\pi_\text{ref}$. Without it, the “reward” of a response would be its raw log-probability, which mostly reflects length and fluency rather than quality. You can drop it if you replace the anchor with something else: SimPO uses the length-normalized log-probability with a margin; ORPO adds an SFT term on the chosen response. Both save the memory and compute of the reference model, at the cost of the derivation’s guarantee.

(c) It is right that DPO optimizes, in the ideal limit, the same KL-regularized reward objective as RLHF; that’s the derivation. It is not RL in Chapter 1’s sense because the agent never acts. It never samples a response, never receives evaluative feedback on its own behavior, and its data does not depend on its policy. It is supervised learning on a fixed dataset with a cleverly chosen loss. That is exactly why it is offline, can’t explore, and can’t correct mistakes on responses the dataset doesn’t contain.

I've done this one

Exercise 13.2A DPO step by handPen & paper●○○

Take $\beta = 0.2$ and a pair with $\log\pi_\theta(y_w\mid x) = -8$, $\log\pi_\text{ref}(y_w\mid x) = -9$, $\log\pi_\theta(y_l\mid x) = -10$, $\log\pi_\text{ref}(y_l\mid x) = -9.5$.

(a) Compute both implicit rewards and the margin. Does the implicit reward model rank the pair correctly, and with what probability?

(b) Compute the loss and the gradient weight $\sigma(\hat r_l - \hat r_w)$.

(c) Compute $\partial\mathcal{L}/\partial\log\pi_\theta(y_w\mid x)$ and $\partial\mathcal{L}/\partial\log\pi_\theta(y_l\mid x)$. Which way does a gradient-descent step move each log-prob?

Solution

(a) $\hat r_w = 0.2\,(-8 - (-9)) = 0.2$. $\hat r_l = 0.2\,(-10 - (-9.5)) = -0.1$. Margin $u = 0.3$. It ranks the pair correctly, with $P(y_w\succ y_l) = \sigma(0.3) = 1/(1 + e^{-0.3}) = 1/1.7408 = 0.574$: correct, but not confident.

(b) Loss $= -\log\sigma(0.3) = \log(1 + e^{-0.3}) = \log 1.7408 = 0.554$. Weight $\sigma(-0.3) = 0.7408/1.7408 = 0.426$.

(c) $\mathcal{L} = -\log\sigma(u)$ with $u = \beta(\log\pi_\theta(y_w) - \log\pi_\text{ref}(y_w)) - \beta(\log\pi_\theta(y_l) - \log\pi_\text{ref}(y_l))$. So $\partial\mathcal{L}/\partial\log\pi_\theta(y_w) = -\sigma(-u)\,\beta = -0.426 \times 0.2 = -0.0851$, and $\partial\mathcal{L}/\partial\log\pi_\theta(y_l) = +0.0851$. Gradient descent moves opposite to the gradient: $\log\pi_\theta(y_w)$ goes up, $\log\pi_\theta(y_l)$ goes down, with equal force. (How the parameters change, and so what happens to other responses, depends on the network; see Exercise 13.4.)

I've done this one

Exercise 13.3The full DPO derivationPen & paper●●○

(a) Starting from the objective $\max_\pi \mathbb{E}_{y\sim\pi(\cdot\mid x)}[r(x,y)] - \beta\,\mathrm{KL}(\pi(\cdot\mid x)\Vert\pi_\text{ref}(\cdot\mid x))$ and its solution $\pi^*(y\mid x) = \pi_\text{ref}(y\mid x)e^{r(x,y)/\beta}/Z(x)$ (take it as given from Chapter 10), derive the DPO loss. Show every step, and say exactly where you use the fact that the two responses share a prompt.

(b) Suppose a dataset also contained comparisons of the form “response $y_1$ to prompt $x_1$ is better than response $y_2$ to prompt $x_2$”. Write the Bradley–Terry probability of such a comparison in terms of $\pi^*$ and $\pi_\text{ref}$, and explain why DPO can’t use it.

(c) Bradley–Terry generalizes to rankings of $K$ responses (the Plackett–Luce model): the probability of the ranking $y_1 \succ y_2 \succ \dots \succ y_K$ is $\prod_{k=1}^{K}\frac{e^{r(x,y_k)}}{\sum_{j=k}^{K}e^{r(x,y_j)}}$. Show that $Z(x)$ also cancels here, and write the resulting loss.

Hint

In (a), take logs of the solution to isolate $r$. In (c), substitute $r = \beta\log\frac{\pi^*}{\pi_\text{ref}} + \beta\log Z(x)$ into each factor and factor $e^{\log Z(x)}$ out of the numerator and the denominator.

Solution

(a) Take the log of the solution: $\log\pi^*(y\mid x) = \log\pi_\text{ref}(y\mid x) + r(x,y)/\beta - \log Z(x)$. Multiply by $\beta$ and rearrange:

$$r(x,y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_\text{ref}(y\mid x)} + \beta\log Z(x).$$

Bradley–Terry says $P(y_w\succ y_l\mid x) = \sigma(r(x,y_w) - r(x,y_l))$. Substitute:

$$r(x,y_w) - r(x,y_l) = \beta\log\frac{\pi^*(y_w\mid x)}{\pi_\text{ref}(y_w\mid x)} - \beta\log\frac{\pi^*(y_l\mid x)}{\pi_\text{ref}(y_l\mid x)} + \beta\log Z(x) - \beta\log Z(x).$$

Here we use the shared prompt: both $\log Z$ terms are evaluated at the same $x$, so they cancel. Now parameterize the policy, $\pi^* \to \pi_\theta$, which makes $\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_\text{ref}}$ a reward model whose optimal policy is $\pi_\theta$ itself. The log-likelihood of the dataset under Bradley–Terry is $\sum_{(x,y_w,y_l)}\log\sigma(\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l))$. Maximizing it is minimizing

$$\mathcal{L}_\text{DPO}(\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_\text{ref}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_\text{ref}(y_l\mid x)}\Big)\Big].$$

(b) $P = \sigma\big(r(x_1,y_1) - r(x_2,y_2)\big) = \sigma\Big(\beta\log\frac{\pi^*(y_1\mid x_1)}{\pi_\text{ref}(y_1\mid x_1)} - \beta\log\frac{\pi^*(y_2\mid x_2)}{\pi_\text{ref}(y_2\mid x_2)} + \beta\log\frac{Z(x_1)}{Z(x_2)}\Big)$. The last term is a ratio of sums over all responses to two different prompts. It is not a function of anything you can compute from the policy’s log-probs on the two given responses, so the loss can’t be evaluated. (A separately trained reward model can in principle use cross-prompt comparisons, although Bradley–Terry rewards are only identified up to a per-prompt shift, so those comparisons carry information the within-prompt model doesn’t even represent.)

(c) Write $\hat r_k = \beta\log\frac{\pi^*(y_k\mid x)}{\pi_\text{ref}(y_k\mid x)}$ and $r(x,y_k) = \hat r_k + c$, $c = \beta\log Z(x)$:

$$\frac{e^{r(x,y_k)}}{\sum_{j\ge k}e^{r(x,y_j)}} = \frac{e^{c}e^{\hat r_k}}{e^{c}\sum_{j\ge k}e^{\hat r_j}} = \frac{e^{\hat r_k}}{\sum_{j\ge k}e^{\hat r_j}}.$$

The common factor $e^c$ cancels in every term of the product because all $K$ responses share $x$. The loss is $-\mathbb{E}\Big[\sum_{k=1}^{K}\Big(\hat r_\theta(x,y_k) - \log\sum_{j=k}^{K}e^{\hat r_\theta(x,y_j)}\Big)\Big]$ with $\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_\text{ref}}$. For $K = 2$ this is the DPO loss, since $\hat r_1 - \log(e^{\hat r_1} + e^{\hat r_2}) = \log\sigma(\hat r_1 - \hat r_2)$ and the $k = 2$ term is $0$. (The DPO paper derives this Plackett–Luce version in its appendix.)

I've done this one

Exercise 13.4The gradient, down to the logitsPen & paper●●○

(a) Derive $\nabla_\theta\mathcal{L}_\text{DPO}$ for one pair and show it equals $-\beta\,\sigma(\hat r_l - \hat r_w)\big(\nabla_\theta\log\pi_\theta(y_w\mid x) - \nabla_\theta\log\pi_\theta(y_l\mid x)\big)$.

(b) Now take a tabular policy over a finite set of responses (the demo’s scenarios 1 and 2): $\pi_\theta(y) = \mathrm{softmax}(z)_y$ with one logit $z_y$ per response, and $\pi_\text{ref}$ fixed. Show that $\partial\mathcal{L}/\partial z = -\beta\,\sigma(\hat r_l - \hat r_w)\,(e_w - e_l)$, where $e_y$ is the one-hot vector for response $y$. Which logits move under gradient descent?

(c) In that tabular model with a single pair, show that a gradient step always increases $\pi_\theta(y_w)$, and that the ratio $\pi_\theta(y)/\pi_\theta(y')$ of any two responses outside the pair doesn’t change. So in this model, displacement needs something more. What? (Exercise 13.7 and the demo’s scenario 3 give two answers.)

Hint

For (b), use $\nabla_z\log\mathrm{softmax}(z)_y = e_y - \pi$. For (c), the step changes $z_w$ by $+\delta$ and $z_l$ by $-\delta$ for some $\delta \gt 0$; differentiate $\log\pi(y_w)$ with respect to $\delta$.

Solution

(a) Let $u = \hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)$ and $\mathcal{L} = -\log\sigma(u)$. Then $\frac{d\mathcal{L}}{du} = -\frac{\sigma'(u)}{\sigma(u)} = -\frac{\sigma(u)(1-\sigma(u))}{\sigma(u)} = -(1-\sigma(u)) = -\sigma(-u)$. The reference terms don’t depend on $\theta$, so $\nabla_\theta u = \beta\big(\nabla_\theta\log\pi_\theta(y_w\mid x) - \nabla_\theta\log\pi_\theta(y_l\mid x)\big)$. By the chain rule, $\nabla_\theta\mathcal{L} = -\sigma(-u)\,\beta\,(\dots)$, and $\sigma(-u) = \sigma(\hat r_l - \hat r_w)$.

(b) $\nabla_z\log\pi(y_w) - \nabla_z\log\pi(y_l) = (e_w - \pi) - (e_l - \pi) = e_w - e_l$: the normalizer’s gradient $\pi$ cancels in the difference. So $\partial\mathcal{L}/\partial z = -\beta\sigma(\hat r_l - \hat r_w)(e_w - e_l)$. Only two logits move: $z_w$ up and $z_l$ down, by equal amounts. All other logits stay put.

(c) After the step, $z_w \to z_w + \delta$, $z_l \to z_l - \delta$ with $\delta \gt 0$. Then $\log\pi(y_w) = z_w + \delta - \log\big(e^{z_w+\delta} + e^{z_l-\delta} + \sum_{y\ne w,l}e^{z_y}\big)$, and

$$\frac{d\log\pi(y_w)}{d\delta} = 1 - \big(\pi(y_w) - \pi(y_l)\big) \gt 0,$$

since $\pi(y_w) - \pi(y_l) \lt 1$. So $\pi(y_w)$ always rises. For $y, y'$ outside the pair, $\pi(y)/\pi(y') = e^{z_y - z_{y'}}$, and neither logit moves, so the ratio is unchanged (both probabilities shrink by the same factor as $y_w$ grows). Displacement needs either several pairs that share responses, so one response is pushed up by one pair and down by another (Exercise 13.7, demo scenario 2), or shared parameters, so moving one response’s logit moves others (demo scenario 3, a real network).

I've done this one

Exercise 13.5Deterministic preferences push the margin to infinityPen & paper●●●

Fix one pair $(x, y_w, y_l)$ and let $p$ be the fraction of annotators who prefer $y_w$. Write $\Delta = h_\theta(x,y_w) - h_\theta(x,y_l)$ for the raw log-ratio margin, where $h_\theta = \log\frac{\pi_\theta}{\pi_\text{ref}}$.

(a) Write the expected DPO loss on this pair as a function of $\Delta$ and find its minimizer. What happens as $p \to 1$? Does $\beta$ stop this?

(b) If the labels come from Bradley–Terry with true reward $r$, so $p = \sigma(r(x,y_w) - r(x,y_l))$, show the minimizer satisfies $\beta\Delta = r(x,y_w) - r(x,y_l)$. What does that say about the policy DPO converges to when every pair of responses is labeled this way (demo scenario 1)?

(c) The expected IPO loss is $p\,\big(\Delta - \tfrac{1}{2\beta}\big)^2 + (1-p)\big(-\Delta - \tfrac{1}{2\beta}\big)^2$. Find its minimizer and evaluate it at $p = 1$.

(d) A real dataset labels each pair once. Explain why (a) is the relevant case, even when annotators disagree a lot.

Hint

Write $u = \beta\Delta$. Use $\frac{d}{du}\log\sigma(u) = \sigma(-u)$ and $\frac{d}{du}\log\sigma(-u) = -\sigma(u)$. For (b), think about which policies have $\beta h_\theta(y) = r(y) + \text{const}$.

Solution

(a) With $u = \beta\Delta$: $\ell(u) = -p\log\sigma(u) - (1-p)\log\sigma(-u)$. Then $\ell'(u) = -p\,\sigma(-u) + (1-p)\,\sigma(u) = -p(1 - \sigma(u)) + (1-p)\sigma(u) = \sigma(u) - p$. It vanishes at $\sigma(u) = p$, i.e. $u^* = \log\frac{p}{1-p}$, and $\ell'' = \sigma(u)(1-\sigma(u)) \gt 0$, so this is the minimum. So $\Delta^* = \frac{1}{\beta}\log\frac{p}{1-p}$. As $p \to 1$, $u^* \to \infty$. At $p = 1$, $\ell'(u) = \sigma(u) - 1 \lt 0$ for every $u$: the loss decreases forever and has no minimizer. $\beta$ doesn’t help: for any $\beta$, $\Delta$ must go to infinity. A larger $\beta$ makes the loss saturate sooner in $\Delta$, but the infimum is still only approached as $\Delta \to \infty$. Because $\pi_\theta(y_w) \le 1$, the chosen log-ratio is capped at $-\log\pi_\text{ref}(y_w)$, so $\Delta \to \infty$ forces $\pi_\theta(y_l) \to 0$.

(b) With $p = \sigma(\Delta r)$, where $\Delta r = r(x,y_w) - r(x,y_l)$, the minimizer is $u^* = \log\frac{\sigma(\Delta r)}{1-\sigma(\Delta r)} = \log\frac{\sigma(\Delta r)}{\sigma(-\Delta r)} = \log e^{\Delta r} = \Delta r$. So $\beta\Delta^* = \Delta r$. If this holds for every pair of responses, then $\beta h_\theta(y) - r(y)$ is the same constant for all $y$, i.e. $\pi_\theta(y) \propto \pi_\text{ref}(y)e^{r(y)/\beta}$: the policy is the RLHF optimum $\pi^*$. (In a tabular model all pairwise conditions can be met at once, and that’s what the demo’s scenario 1 shows.)

(c) $\frac{d}{d\Delta}$: $2p(\Delta - c) - 2(1-p)(-\Delta - c) = 2\Delta - 2c(2p - 1)$ with $c = \frac{1}{2\beta}$. Setting it to zero gives $\Delta^* = \frac{2p - 1}{2\beta}$, a finite value for every $p$ (the loss is a convex quadratic). At $p = 1$, $\Delta^* = \frac{1}{2\beta}$: the larger $\beta$, the smaller the target margin, so $\beta$ acts as a leash again. In the demo’s implicit-reward units, $\beta\Delta^* = \frac12$, the value where IPO’s margin stops.

(d) With one label per pair, the empirical $p$ of every pair is exactly 1 (or 0). Minimizing the empirical loss is minimizing a sum of $p = 1$ terms, each of which wants an infinite margin, whatever the true annotator agreement was. Only early stopping, a small learning rate, and the model’s limited capacity keep the margins finite. This is Azar et al.’s argument for IPO.

I've done this one

Exercise 13.6The two limits of βPen & paper●●○

(a) For fixed $\theta$, expand the DPO loss of one pair to first order in $\beta$ around $\beta = 0$. What does the gradient become, and why might training on it for a long time be dangerous? (The DPO paper reports what happens.)

(b) As $\beta \to \infty$, what happens to the gradient weight $\sigma(\hat r_l - \hat r_w)$ once the policy has moved even slightly in the right direction, and what policy does training end at?

(c) Connect both answers to the optimum $\pi^* \propto \pi_\text{ref}e^{r/\beta}$: which policy does DPO target in each limit?

Hint

Write the loss as $-\log\sigma(\beta\Delta)$ with $\Delta$ the raw log-ratio margin, and Taylor-expand $\log(1 + e^{-u})$ around $u = 0$. For (b), the weight is $\sigma(-\beta\Delta)$: what does it do for a fixed small $\Delta \gt 0$ as $\beta$ grows?

Solution

(a) With $u = \beta\Delta$ and $-\log\sigma(u) = \log(1 + e^{-u}) = \log 2 - \frac{u}{2} + O(u^2)$, the loss is $\log 2 - \frac{\beta}{2}\Delta + O(\beta^2)$. Its gradient is $-\frac{\beta}{2}\big(\nabla\log\pi_\theta(y_w) - \nabla\log\pi_\theta(y_l)\big)$: every pair gets the same weight $\frac12$, forever, regardless of how well the pair is already ranked. This is the “naive” objective (raise the winner, lower the loser, with no stopping point) that the DPO paper reports can make the model degenerate: the log-probability of the loser can be pushed toward $-\infty$, dragging shared tokens and probability mass with it.

(b) The weight is $\sigma(-\beta\Delta)$. For any fixed $\Delta \gt 0$, as $\beta\to\infty$ this goes to $0$. So as soon as the policy ranks a pair correctly by any tiny log-ratio margin, the pair stops contributing. Training ends with the policy almost exactly at $\pi_\text{ref}$, just nudged enough to get the orderings right.

(c) In the target $\pi^* \propto \pi_\text{ref}e^{r/\beta}$: $\beta\to\infty$ gives $\pi^*\to\pi_\text{ref}$ (the leash is infinitely tight), matching (b). $\beta\to 0$ gives $\pi^*$ concentrated on the highest-reward response (no leash), which requires infinite log-ratios, matching the unbounded push in (a). In both limits, the loss’s behavior and the target’s behavior agree.

I've done this one

Exercise 13.7Likelihood displacement with two pairsPen & paper●●○

A tabular policy over three responses A, B, C starts at the uniform reference, $\pi_\text{ref} = (\frac13, \frac13, \frac13)$, with logits $z = (0,0,0)$. The dataset has two pairs, each labeled once: B ≻ A and C ≻ B. B is the chosen response in the first pair.

(a) Using Exercise 13.4(b), write the gradient of the total loss (the mean over the two pairs) with respect to $z$ at the start. Which logits move?

(b) Argue that throughout gradient descent the two margins stay equal, so $z = (-\delta, 0, \delta)$ for some growing $\delta \ge 0$. Write $\log\pi_\theta(B)$ as a function of $\delta$ and evaluate it at $\delta = 0, 1, 2$.

(c) Training reports 100% reward accuracy and growing margins. What happened to the “chosen” response B, and where did its probability go?

Hint

Because $\pi_\text{ref}$ is uniform, each log-ratio margin is just a difference of logits, e.g. $z_B - z_A$. Pair 1 pushes $z_B$ up and pair 2 pushes it down; when are the two pushes equal? For (b), $\pi_\theta(B) = e^{z_B}/\sum_y e^{z_y}$.

Solution

(a) Both margins are 0 at the start, so both weights are $\beta\sigma(0) = \beta/2$. The gradient of the mean loss is $-\frac12\cdot\frac{\beta}{2}\big[(e_B - e_A) + (e_C - e_B)\big] = -\frac{\beta}{4}(e_C - e_A)$. Gradient descent raises $z_C$, lowers $z_A$ by the same amount, and leaves $z_B$ unchanged: the push up from pair 1 and the push down from pair 2 cancel.

(b) Suppose $z = (-\delta, 0, \delta)$. The two margins are $\beta(z_B - z_A) = \beta\delta$ and $\beta(z_C - z_B) = \beta\delta$, equal, so the two weights are equal, and by the same calculation as in (a) the gradient is again proportional to $e_C - e_A$. So the form $(-\delta, 0, \delta)$ is preserved, with $\delta$ increasing. Then

$$\log\pi_\theta(B) = 0 - \log\big(e^{-\delta} + 1 + e^{\delta}\big) = -\log(1 + 2\cosh\delta).$$

At $\delta = 0$: $-\log 3 = -1.099$. At $\delta = 1$: $-\log(1 + 3.086) = -1.408$. At $\delta = 2$: $-\log(1 + 7.524) = -2.143$. It decreases monotonically in $\delta$.

(c) Both pairs are ranked correctly, with growing margins, yet B, the winner of the first pair, falls from $\frac13$ to $0.117$ at $\delta = 2$, and on toward 0. Its probability went to C (and A’s too): $\pi(C) = e^{2}/8.524 = 0.867$ at $\delta = 2$. A response can be “chosen” in the data and still lose probability, because the loss only sees margins. In this example the outcome is reasonable (C beat B, so C is the better answer). With shared parameters the mass can go somewhere unreasonable instead, which is scenario 3 in the demo.

I've done this one

Exercise 13.8Explore the demoExplore the demo●○○

(a) Scenario 1, DPO, β = 1. Train to the end. Check π_θ, the implicit rewards and the mean margin against the worked example in “The trick, with numbers”. Then set β = 0.3 and train again. What changes, and what stays the same?

(b) Scenario 2, DPO. Compare β = 0.1 and β = 1. Where does each end up? Then switch to IPO with β = 1 and compare with π*. Which response is never learned about, and why?

(c) Scenario 3, DPO, β = 1. Describe the curve of log π_θ(“No.”). Which response wins, and how does its true reward compare with the others’? Now try DPO + NLL, then IPO. Which one fixes the problem?

Solution

(a) At β = 1, π_θ converges to (0.179, 0.292, 0.529), exactly the π* ticks. The implicit rewards converge to (−1.027, −0.027, 0.973), which is $r - \beta\log Z$ with $\beta\log Z = 1.027$, the table in the text. The mean margin over the three pairs converges to (1 + 1 + 2)/3 = 1.333, the mean true reward gap, and KL(π_θ ‖ π*) goes to 0. At β = 0.3, π* is much sharper (C gets most of the mass), and DPO follows it. But the implicit rewards still differ by 1 and 2, and the mean margin is still 1.333. DPO learns the reward differences, whatever β, and β only decides how strongly the policy acts on them.

(b) Both go to “all mass on C”, with B and D falling too. With deterministic labels, β changes the speed (how long the curves take) but not the destination, as Exercise 13.5 showed. The margin keeps growing. With IPO at β = 1 the margin stops at 0.5 and the policy stays close to the reference: about (0.25, 0.31, 0.34, 0.10). In both cases D, which has the highest true reward, is never learned about: it appears in no pair, so no gradient ever targets its logit, and it only loses probability through normalization. π* gives D the most mass. The data simply never asked about it: this is DPO being offline.

(c) log π_θ(“No.”) first rises for a few dozen steps, then falls steadily, while “Yes.” (true reward −3, by far the worst) rises to over 90% of the mass. DPO + NLL fixes it: “No.” goes toward probability 1, because the NLL term pulls θ toward the chosen response’s features while DPO separates it from “Never.”. IPO limits the damage because the margin target is finite, but at β = 1 “Yes.” still about doubles its probability (0.05 → 0.10): with a feature-based policy, any movement along the direction φ(No.) − φ(Never.) helps “Yes.”.

I've done this one

Exercise 13.9Lab, Tasks 1–3: the lossesCode●●○

In dpo.py, implement sequence_logprob (Task 1), implicit_reward and dpo_loss (Task 2), then ipo_loss and simpo_loss (Task 3). The tests check the worked example from this chapter ($\beta = 0.1$, log-probs $-12, -15, -13, -14$, loss $0.598$) and the gradient $\mp\beta\,\sigma(\hat r_l - \hat r_w)$, IPO’s zero at margin $1/(2\beta)$, and numerical stability at a margin of $-500$.

Hint

Multiply the per-token log-probs by the mask before summing. For the losses use torch.nn.functional.logsigmoid; torch.log(torch.sigmoid(u)) underflows to -inf for very negative u.

Solution

The reference solution (solutions/dpo.py) is a few lines per function:

def sequence_logprob(token_logp, mask, average=False):
    total = (token_logp * mask).sum(dim=-1)
    return total / mask.sum(dim=-1).clamp(min=1.0) if average else total

def implicit_reward(policy_logp, ref_logp, beta):
    return beta * (policy_logp - ref_logp)

def dpo_loss(pw, pl, rw, rl, beta):
    r_w, r_l = implicit_reward(pw, rw, beta), implicit_reward(pl, rl, beta)
    return -F.logsigmoid(r_w - r_l).mean(), r_w.detach(), r_l.detach()

def ipo_loss(pw, pl, rw, rl, beta):
    h = (pw - rw) - (pl - rl)
    return ((h - 1.0 / (2.0 * beta)) ** 2).mean()

def simpo_loss(avg_w, avg_l, beta, margin):
    return -F.logsigmoid(beta * avg_w - beta * avg_l - margin).mean()

Key points: the mask multiplication comes before the sum (padding positions can hold anything); IPO’s $h$ has no $\beta$ in front, while its target does; F.logsigmoid is computed stably as $\min(u,0) - \log(1 + e^{-\lvert u\rvert})$; and the implicit rewards are returned detached because they are for logging only.

I've done this one

Exercise 13.10Lab, Task 4 and the experiment: when DPO makes the model worseCode●●○

Implement dpo_step (Task 4): policy log-probs with gradients, reference log-probs under torch.no_grad(), the chosen loss, an optional NLL term on the chosen response, one optimizer step, and the diagnostics dict. Then run python run.py, which builds about 2,200 offline pairs (a correct answer preferred to a wrong one, both sampled from the SFT TinyLM) and trains four variants on them.

(a) Which variant lowers answer accuracy? What happens to its held-out $\log\pi_\theta(\text{chosen})$, and how is that possible when its reward accuracy on training pairs is above 90%?

(b) Explain why β = 1 behaves so differently from β = 0.1 here, using the gradient weight $\sigma(-u)$.

(c) Which of the three fixes in the run correspond to which section of the chapter?

Hint

The loss stops pushing a pair once the implicit-reward margin $u = \beta\Delta$ is a few units. How large must the raw log-ratio margin $\Delta$ be for that, at each β?

Solution

The core of the reference dpo_step:

tok_w, mask_w = response_logprobs(policy, prompts, chosen)
tok_l, mask_l = response_logprobs(policy, prompts, rejected)
pi_w, pi_l = sequence_logprob(tok_w, mask_w), sequence_logprob(tok_l, mask_l)
with torch.no_grad():
    ref_w = sequence_logprob(*response_logprobs(ref, prompts, chosen))
    ref_l = sequence_logprob(*response_logprobs(ref, prompts, rejected))
loss, r_w, r_l = dpo_loss(pi_w, pi_l, ref_w, ref_l, cfg.beta)        # or ipo / simpo
if cfg.nll_coef > 0:
    loss = loss - cfg.nll_coef * sequence_logprob(tok_w, mask_w, average=True).mean()
opt.zero_grad(); loss.backward(); opt.step()

followed by the stats: (r_w > r_l).float().mean(), (r_w - r_l).mean(), pi_w.mean(), pi_l.mean() as Python floats.

(a) With the reference solution, DPO β = 0.1 drops greedy accuracy from 0.76 to about 0.38, and its held-out $\log\pi_\theta(\text{chosen})$ falls from −0.67 to about −2.5, while $\log\pi_\theta(\text{rejected})$ falls much further (to about −9.8). The margin, which is all the loss sees, keeps growing, so the ranking is right; but both responses lose probability, and the freed mass goes to other wrong answers that appear in no pair (for example, the model starts answering 41+19= with 71). This is likelihood displacement: the right answer is “preferred” and less likely at the same time.

(b) A pair stops contributing once $\sigma(-u)$ is small, say $u \approx 3$ (weight 0.05). With β = 1 that needs a raw log-ratio margin $\Delta \approx 3$ nats (natural-log units), which the model reaches quickly; then the gradients fade and the policy stays near the reference. With β = 0.1 it needs $\Delta \approx 30$ nats. The optimizer keeps pushing hard on every pair for many epochs over the same small dataset, and the cheapest way to grow $\Delta$ is to crush the rejected answers and everything that shares parameters with them, which includes the correct answers’ neighbors and the correct answers themselves. (On a dataset this small, with this many epochs, a small β is a weak leash, as the “deterministic preferences” section predicts.)

(c) DPO β = 1: a tighter leash, the “deterministic preferences” section (and Exercise 13.6). DPO β = 0.1 + NLL: the likelihood-displacement fix (Llama 3, Pang et al.); it does best here, about 0.91 greedy accuracy, with $\log\pi_\theta(\text{chosen})$ rising to −0.18. IPO β = 0.1: the finite-target fix from the variants section; about 0.84 greedy accuracy.

I've done this one

Further reading

• Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The original paper. Section 4 is the derivation, section 5 the theory (equivalence classes), and the appendix has the Plackett–Luce version and the gradient analysis.
• Azar et al. (2023), A General Theoretical Paradigm to Understand Learning from Human Preferences. The ΨPO framework and IPO; the clearest statement of why deterministic preferences break DPO’s regularization.
• Razin et al. (2025), Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. Why chosen log-probs fall, the “No”/“Never”/“Yes” example, and a data-filtering fix.
• Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. The theorem that DPO’s solutions include PPO’s and more, plus the PPO tuning that made it win.
• Ivison et al. (2024), Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. A careful controlled comparison; the lesson that data quality matters most.
• Meng, Xia & Chen (2024), SimPO: Simple Preference Optimization with a Reference-Free Reward, and Ethayarajh et al. (2024), KTO: Model Alignment as Prospect Theoretic Optimization. The two most widely used variants, each well written.
• Guo et al. (2024), Direct Language Model Alignment from Online AI Feedback, and Tajwar et al. (2024), Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. Why on-policy data matters, from two angles.
• Lambert et al. (2024), Tülu 3: Pushing Frontiers in Open Language Model Post-Training. A fully open recipe with a DPO stage, its hyperparameters, and DPO-vs-PPO-vs-SimPO comparisons on the same data.

Next, Chapter 14 asks where the preferences and rewards come from in the first place: human labelers, AI feedback and constitutions, LLM judges, rubrics, and verifiers.