Trust regions: TRPO and PPO

In supervised learning a step that is too big is a nuisance. The loss jumps, the next few steps pull it back, and training goes on, because the dataset never changed. Reinforcement learning has no such safety net. The policy produces its own data. Take one step that makes the policy much worse, and the next batch is collected by that worse policy. It may never visit the states, or try the actions, that would show it its mistake. One bad step can end a training run.

The actor-critic from Chapter 8 has a second, quieter problem: it is wasteful. It collects a batch of experience (the expensive part), takes one gradient step, and throws the batch away. This chapter fixes both problems with a single idea: reuse each batch for many gradient steps, but never let the policy’s behaviour move far from the policy that collected the batch. TRPO makes that idea exact with a constraint. PPO gets most of the benefit with one min and a clip, which is why PPO became the default deep-RL algorithm and, later, the optimizer of RLHF.

Why one big step can ruin everything

A step in θ is not a step in behaviour

Start with the smallest possible policy: one state, two actions, and one parameter.

$$\pi_\theta(\text{right}) = \sigma(\theta) = \frac{1}{1 + e^{-\theta}}, \qquad \pi_\theta(\text{left}) = 1 - \sigma(\theta).$$

Take the same parameter step, $\Delta\theta = 1$, from two different starting points, and measure how much the behaviour changed with the KL divergence between the old and new action distributions:

Step	$\pi(\text{right})$ before → after	$\mathrm{KL}(\pi_\text{old} \,\Vert\, \pi_\text{new})$
$\theta: 0 \to 1$	0.500 → 0.731	0.120
$\theta: 4 \to 5$	0.982 → 0.993	0.0066

The same step in parameter space is an 18 times larger change in behaviour at $\theta = 0$ than at $\theta = 4$. Nothing about the size of $\Delta\theta$ tells you how far the policy moved. With a neural network of a million parameters this gets worse: some directions barely change anything, and others change the action probabilities in thousands of states at once. A learning rate is a step size in the wrong space.

You can’t undo a bad step

Now suppose a noisy advantage estimate tells you “right” is great, and a large step pushes $\pi(\text{right})$ from 0.5 to 0.999. If “left” was actually better, how do you find out? Only by trying “left”, which the policy now does once in a thousand steps. And the gradient that could push back is small too: from Chapter 7, $\partial \log \pi(\text{left}) / \partial\theta = -\sigma(\theta)$, and since “left” is sampled with probability $1-\sigma(\theta)$, its expected contribution scales with $\sigma(1-\sigma) \approx 0.001$. The policy is stuck: it has stopped exploring the very action that would reveal the mistake. This is policy collapse, and in deep RL it shows up as a learning curve that climbs, falls off a cliff, and stays at the bottom.

And you waste data

The obvious fix for wasted data is to run several gradient steps on each batch. But the policy-gradient estimate $\hat{\mathbb E}_t[\nabla_\theta \log\pi_\theta(a_t \mid s_t)\,\hat A_t]$ is only correct when the actions were sampled from the current policy. After the first step they weren’t: they came from the old one. Keep stepping on the same batch and you are following a gradient of the wrong thing (Exercise 9.1 makes this concrete).

Reusing a batch: importance sampling

The trick

You want an average under one distribution, $p$, but your samples come from another, $q$. Reweight each sample by how much more (or less) likely it is under $p$:

$$\mathbb E_{a \sim p}[f(a)] = \sum_a p(a) f(a) = \sum_a q(a)\,\frac{p(a)}{q(a)}\, f(a) = \mathbb E_{a\sim q}\!\left[\frac{p(a)}{q(a)}\, f(a)\right].$$

This is exact as long as $q(a) > 0$ wherever $p(a) > 0$. The weight $p/q$ is the importance ratio. For a policy, the “old” policy $\pi_{\theta_\text{old}}$ (written $\pi_\text{old}$ from now on) collected the data and the current $\pi_\theta$ is the one we want to evaluate. The ratio at time step $t$ is

$$\rho_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_\text{old}(a_t \mid s_t)}.$$

At the start of an update, $\theta = \theta_\text{old}$ and every $\rho_t = 1$. As $\theta$ moves, actions the new policy likes more get $\rho_t > 1$ and actions it likes less get $\rho_t < 1$. (The PPO paper writes $r_t(\theta)$; we use $\rho_t$ so it doesn’t clash with the reward $r_t$.)

The surrogate objective

Apply the trick to the policy-improvement question “if I switched from $\pi_\text{old}$ to $\pi_\theta$, how much better would I do?” and you get the surrogate objective

$$L(\theta) = \hat{\mathbb E}_t\big[\rho_t(\theta)\,\hat A_t\big],$$

where $\hat{\mathbb E}_t$ is the average over all time steps in the batch and $\hat A_t$ is the advantage estimate computed once, with the old policy and critic (GAE from Chapter 8). In words: take every action in the batch, ask how much more likely the new policy makes it, and weight that by how good the action turned out to be. Raising the probability of good actions and lowering the probability of bad ones increases $L$.

Two facts make $L$ the right thing to optimize near $\theta_\text{old}$:

• It starts at zero. At $\theta = \theta_\text{old}$, $L = \hat{\mathbb E}_t[\hat A_t]$, which is about zero, because advantages average to zero under the policy that defines them.
• Its gradient there is the policy gradient. Since $\nabla_\theta \rho_t = \rho_t \nabla_\theta \log\pi_\theta(a_t\mid s_t)$ and $\rho_t = 1$ at $\theta_\text{old}$,

$$\nabla_\theta L(\theta)\Big|_{\theta_\text{old}} = \hat{\mathbb E}_t\big[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\,\hat A_t\big],$$

exactly the actor-critic gradient you already know. So the first gradient step on $L$ is a vanilla policy-gradient step. The difference is what happens on the second, third, and tenth steps: $L$ keeps tracking (an estimate of) the improvement of the new policy, using data from the old one.

How good is $L$ as a stand-in for the true objective $J(\theta)$? The next derivation shows that the two agree in value and slope at $\theta_\text{old}$ and differ in exactly one place: which states the average is taken over.

∂ The performance difference lemma, and why L matches J to first order

Step 1: the performance difference lemma (Kakade and Langford, 2002). For any two policies $\pi$ and $\pi'$ with the same start-state distribution,

$$J(\pi') - J(\pi) = \mathbb E_{\tau\sim\pi'}\Big[\sum_{t} \gamma^t A^{\pi}(s_t, a_t)\Big].$$

Proof. The advantage is the expected one-step TD error: $A^\pi(s_t, a_t) = \mathbb E[r_t + \gamma V^\pi(s_{t+1}) - V^\pi(s_t) \mid s_t, a_t]$, where the expectation is over the environment’s next state. The environment’s dynamics don’t depend on the policy, so this inner expectation is the same along a trajectory of $\pi'$. Hence

$$\mathbb E_{\tau\sim\pi'}\Big[\sum_t \gamma^t A^\pi(s_t,a_t)\Big] = \mathbb E_{\tau\sim\pi'}\Big[\sum_t \gamma^t r_t\Big] + \mathbb E_{\tau\sim\pi'}\Big[\sum_t \big(\gamma^{t+1}V^\pi(s_{t+1}) - \gamma^t V^\pi(s_t)\big)\Big].$$

The second sum telescopes to $-V^\pi(s_0)$ (with $V^\pi = 0$ at a terminal state, or $\gamma^t V^\pi \to 0$ for an infinite horizon). So the right-hand side is $J(\pi') - \mathbb E_{s_0}[V^\pi(s_0)] = J(\pi') - J(\pi)$. ∎

Step 2: rewrite with importance weights. Write $p_t^{\theta}(s)$ for the probability that $\pi_\theta$ is in state $s$ at time $t$. Applying the lemma with $\pi = \pi_\text{old}$, $\pi' = \pi_\theta$:

$$J(\theta) - J(\theta_\text{old}) = \sum_t \gamma^t \sum_s p_t^{\theta}(s)\, f_\theta(s), \qquad f_\theta(s) = \sum_a \pi_\theta(a\mid s)\,A^{\pi_\text{old}}(s,a) = \mathbb E_{a\sim\pi_\text{old}}\big[\rho(\theta)\,A^{\pi_\text{old}}(s,a)\big].$$

The actions can be importance-weighted, because we know both policies’ probabilities. The states can’t: $p_t^\theta$ depends on the environment’s dynamics, which we don’t know.

Step 3: the surrogate. Replace the unknown new-policy state distribution by the old one:

$$L(\theta) = \sum_t \gamma^t \sum_s p_t^{\theta_\text{old}}(s)\, f_\theta(s) = \mathbb E_{\tau\sim\pi_\text{old}}\Big[\sum_t \gamma^t \rho_t(\theta)\, A^{\pi_\text{old}}(s_t,a_t)\Big].$$

Step 4: first-order agreement. At $\theta_\text{old}$, $f_{\theta_\text{old}}(s) = \mathbb E_{a\sim\pi_\text{old}}[A^{\pi_\text{old}}(s,a)] = 0$ for every $s$ (the advantage averages to zero under its own policy). So both $J(\theta) - J(\theta_\text{old})$ and $L(\theta)$ are zero at $\theta_\text{old}$. For the gradient, use the product rule on the true difference:

$$\nabla_\theta\big[J(\theta) - J(\theta_\text{old})\big] = \sum_t \gamma^t \sum_s \Big[\nabla_\theta p_t^\theta(s)\, f_\theta(s) + p_t^\theta(s)\,\nabla_\theta f_\theta(s)\Big].$$

At $\theta_\text{old}$ the first term vanishes because $f_{\theta_\text{old}} = 0$, and the second term is exactly $\nabla_\theta L(\theta_\text{old})$. So $J$ and $L$ have the same value and the same gradient at $\theta_\text{old}$. The change in the state distribution only matters at second order, which is why $L$ is trustworthy only near $\theta_\text{old}$.

In practice we estimate $L$ by $\hat{\mathbb E}_t[\rho_t \hat A_t]$: an average over all time steps in the batch, with $\hat A_t$ from GAE. Implementations drop the $\gamma^t$ weighting of the state distribution. This is a standard simplification, slightly biased, and used by essentially every PPO implementation.

Why you can’t trust the surrogate far away

Let’s watch $L$ mislead us on a problem small enough to do by hand. A 3-armed bandit has true mean rewards A = 1.0, B = 0.6, C = 0.0 (unknown to the learner), plus noise. The old policy is uniform, $\pi_\text{old} = (\tfrac13, \tfrac13, \tfrac13)$. We draw a batch of four pulls:

pull	arm	reward	advantage $\hat A$ (reward minus batch mean 0)
1	A	0.5	+0.5
2	B	1.5	+1.5
3	B	0.5	+0.5
4	C	−2.5	−2.5

With a bandit, every ratio is $\rho = \pi_\theta(a)/\tfrac13 = 3\pi_\theta(a)$, so

$$L = \tfrac14\big[3\pi(A)(0.5) + 3\pi(B)(1.5 + 0.5) + 3\pi(C)(-2.5)\big] = 0.375\,\pi(A) + 1.5\,\pi(B) - 1.875\,\pi(C).$$

The surrogate is linear in the probabilities. A linear function on the probability simplex is maximized at a corner: put all the probability on B, and $L = 1.5$. So if you optimize $L$ to convergence on this batch, you get a deterministic policy that always pulls B. Its true value is $J = 0.6$, worse than arm A’s 1.0, and $\mathrm{KL}(\pi_\text{old}\,\Vert\,\pi) = \infty$. Worse, it will now never pull A again, so it can never discover the mistake.

What went wrong? Arm B had two lucky draws. For any fixed $\theta$ the surrogate is a (nearly) unbiased estimate of the improvement, but as $\pi_\theta$ moves away from $\pi_\text{old}$ the ratios grow and the estimate leans harder and harder on a few samples. Optimizing a noisy estimate to its maximum finds the noise.

In an MDP there is a second reason, from the derivation above: $L$ averages over the states the old policy visits. A very different policy visits different states, where the batch has nothing to say.

Trust regions and TRPO

Measure the step in behaviour: KL

The natural yardstick for “how far did the policy move” is the KL divergence between old and new action distributions, averaged over the states in the batch:

$$\overline{\mathrm{KL}}(\theta) = \hat{\mathbb E}_{s}\Big[\mathrm{KL}\big(\pi_\text{old}(\cdot\mid s)\,\Vert\,\pi_\theta(\cdot\mid s)\big)\Big], \qquad \mathrm{KL}(p\,\Vert\,q) = \sum_a p(a)\log\frac{p(a)}{q(a)}.$$

It doesn’t care how the policy is parameterized: a 2-parameter softmax and a 7-billion-parameter transformer that produce the same action probabilities have the same KL. This is exactly what the learning rate couldn’t give us.

The trust-region problem is then:

$$\max_\theta\; L(\theta) \quad\text{subject to}\quad \overline{\mathrm{KL}}(\theta) \le \delta,$$

with $\delta$ a small number such as 0.01. There is a theorem behind this choice. The TRPO paper (Schulman et al., 2015, building on Kakade and Langford’s conservative policy iteration) proves the lower bound

$$J(\theta) \;\ge\; J(\theta_\text{old}) + L(\theta) - C\,\max_s \mathrm{KL}\big(\pi_\text{old}(\cdot\mid s)\,\Vert\,\pi_\theta(\cdot\mid s)\big), \qquad C = \frac{4\gamma\,\max_{s,a}\lvert A^{\pi_\text{old}}(s,a)\rvert}{(1-\gamma)^2},$$

where $L$ here is the exact surrogate from the derivation (with the true advantage). The right-hand side equals $J(\theta_\text{old})$ at $\theta_\text{old}$ and lies below $J$ everywhere else, so maximizing it can only improve $J$: a minorize–maximize scheme, the same logic as EM or a variational lower bound on a free energy. In practice $C$ is so large that the steps it allows are tiny, so TRPO swaps the penalty for the constraint $\overline{\mathrm{KL}} \le \delta$ and treats $\delta$ as a hyperparameter.

The Fisher metric

For small steps, KL is a quadratic form in $\Delta\theta = \theta - \theta_\text{old}$:

$$\mathrm{KL}\big(\pi_{\theta_\text{old}}\,\Vert\,\pi_{\theta_\text{old}+\Delta\theta}\big) \approx \tfrac12\,\Delta\theta^\top F\,\Delta\theta, \qquad F = \mathbb E_{s,\;a\sim\pi_\text{old}}\Big[\nabla_\theta\log\pi_\theta(a\mid s)\,\nabla_\theta\log\pi_\theta(a\mid s)^\top\Big].$$

$F$ is the Fisher information matrix. If you know some differential geometry, this is the familiar picture: the space of policies is a curved manifold, $F$ is its metric tensor, and $\tfrac12 \Delta\theta^\top F \Delta\theta$ is the squared line element $\tfrac12 g_{ij}\,d\theta^i d\theta^j$. The plain gradient measures distance with the identity matrix, as if parameter space were flat and every direction counted equally. The Fisher metric measures distance by how much the distribution changes.

∂ Why KL ≈ ½ Δθᵀ F Δθ

Fix a state and let $\ell(\theta) = \log\pi_\theta(a\mid s)$. Then $\mathrm{KL}(\pi_{\theta_\text{old}}\Vert\pi_{\theta_\text{old}+\Delta}) = \mathbb E_{a\sim\pi_\text{old}}[\ell(\theta_\text{old}) - \ell(\theta_\text{old}+\Delta)]$. Taylor-expand $\ell$:

$$\ell(\theta_\text{old}+\Delta) = \ell + \nabla\ell^\top\Delta + \tfrac12\Delta^\top\nabla^2\ell\,\Delta + O(\lVert\Delta\rVert^3),$$

so

$$\mathrm{KL} = -\,\mathbb E[\nabla\ell]^\top\Delta - \tfrac12\,\Delta^\top\,\mathbb E[\nabla^2\ell]\,\Delta + O(\lVert\Delta\rVert^3).$$

First-order term. $\mathbb E_{a\sim\pi}[\nabla\log\pi(a)] = \sum_a \pi(a)\frac{\nabla\pi(a)}{\pi(a)} = \nabla\sum_a\pi(a) = \nabla 1 = 0$. (This is the same identity that makes baselines unbiased in Chapter 7.)

Second-order term. Differentiate $\nabla\ell = \nabla\pi/\pi$ once more: $\nabla^2\ell = \frac{\nabla^2\pi}{\pi} - \nabla\ell\,\nabla\ell^\top$. Taking the expectation, $\mathbb E[\nabla^2\pi/\pi] = \sum_a\nabla^2\pi(a) = \nabla^2 1 = 0$, so

$$-\mathbb E[\nabla^2\ell] = \mathbb E[\nabla\ell\,\nabla\ell^\top] = F.$$

This is the information identity. Putting it together, $\mathrm{KL} \approx \tfrac12\Delta^\top F\Delta$. Averaging over states gives the $F$ in the main text.

Example: the one-parameter policy. For $\pi(\text{right}) = \sigma(\theta)$: $\nabla\log\sigma(\theta) = 1-\sigma$ and $\nabla\log(1-\sigma(\theta)) = -\sigma$. So

$$F = \sigma(1-\sigma)^2 + (1-\sigma)\sigma^2 = \sigma(1-\sigma).$$

At $\theta = 0$, $F = 0.25$; at $\theta = 4$, $F = 0.0177$. The step $\Delta\theta = 1$ therefore costs about $\tfrac12(0.25) = 0.125$ nats of KL at $\theta=0$ and $\tfrac12(0.0177) = 0.0088$ at $\theta=4$, close to the exact 0.120 and 0.0066 in the table at the start of the chapter.

The natural gradient

Now solve the trust-region problem for a small step. Linearize the objective, $L(\theta_\text{old}+\Delta) \approx g^\top\Delta$ with $g = \nabla_\theta L(\theta_\text{old})$ the ordinary policy gradient, and use the quadratic KL:

$$\max_\Delta\; g^\top\Delta \quad\text{subject to}\quad \tfrac12\Delta^\top F\Delta \le \delta.$$

With a Lagrange multiplier $\lambda$: $g - \lambda F\Delta = 0$, so $\Delta = \frac1\lambda F^{-1}g$. Make the constraint tight: $\tfrac12 \lambda^{-2}\,g^\top F^{-1}g = \delta$. Therefore

$$\Delta\theta = \sqrt{\frac{2\delta}{g^\top F^{-1} g}}\;F^{-1}g.$$

The direction $F^{-1}g$ is the natural gradient (Amari, 1998; Kakade, 2001 for policies): steepest ascent when distance is measured by KL instead of by Euclidean distance in $\theta$. It doesn’t depend on how you parameterize the policy (to first order), the same way a covariant derivative doesn’t depend on your coordinates.

Worked example. Back to $\pi(\text{right}) = \sigma(\theta)$, and say “right” pays 1 and “left” pays 0, so $J(\theta) = \sigma(\theta)$ and $g = \sigma(1-\sigma)$. That is the same as $F$, so $F^{-1}g = 1$ for every $\theta$, and the trust-region step is $\Delta\theta = \sqrt{2\delta/F}$. With $\delta = 0.01$:

$\theta$	$\pi(\text{right})$	plain gradient $g$	natural step $\Delta\theta = \sqrt{2\delta/F}$	actual KL of that step
0	0.500	0.250	0.283	0.0100
2	0.881	0.105	0.436	0.0090
4	0.982	0.018	1.064	0.0073

The plain gradient shrinks 14-fold as the policy saturates, so a fixed learning rate crawls exactly where the policy is nearly deterministic. The natural step grows there, because a big parameter move is a small behaviour move. Every step changes the behaviour by about the same KL.

TRPO in practice

For a neural network, $F$ has (number of parameters)² entries, so you can never form it, let alone invert it. Trust Region Policy Optimization (TRPO; Schulman et al., 2015) gets around this with three tricks:

1. Conjugate gradient. Solve $Fx = g$ iteratively. Conjugate gradient only needs matrix–vector products $Fv$, never $F$ itself.
2. Fisher-vector products by double backprop. $Fv$ equals the Hessian of $\overline{\mathrm{KL}}$ at $\theta_\text{old}$ times $v$, which is $\nabla_\theta\big((\nabla_\theta\overline{\mathrm{KL}})^\top v\big)$: two backward passes, no matrices.
3. A line search. The quadratic approximation can be wrong, so shrink the step until the actual KL is within $\delta$ and the surrogate actually improved.

TRPO, one iteration
1. Collect a batch with π_old; compute advantages Â_t (GAE).
2. g ← gradient of L(θ) at θ_old                  (the ordinary policy gradient)
3. x ≈ F⁻¹ g  via ~10 steps of conjugate gradient, using only products F·v
4. Δ ← sqrt(2δ / (xᵀ F x)) · x                    (the natural-gradient step)
5. for j = 0, 1, 2, ...:                           (backtracking line search)
       θ ← θ_old + 0.5^j · Δ
       if mean KL(π_old ‖ π_θ) ≤ δ and L(θ) > 0: accept θ and stop
6. Fit the critic V_φ by regression on the returns (a separate optimizer).

TRPO works and comes with a theory. But it is complicated: second-order machinery, a line search, and awkwardness when the policy and value function share a network (the constraint applies to the policy only) or when the network has noise like dropout. People wanted something with TRPO’s stability that is just “a loss you hand to Adam.”

PPO-clip: a trust region from one min

Proximal Policy Optimization (PPO; Schulman et al., 2017) drops the constraint. Instead, it changes the objective so that moving the ratio far from 1 stops paying. For each sample:

$$L^{\text{CLIP}}(\theta) = \hat{\mathbb E}_t\Big[\min\big(\rho_t(\theta)\,\hat A_t,\;\; \operatorname{clip}(\rho_t(\theta),\,1-\epsilon,\,1+\epsilon)\,\hat A_t\big)\Big],$$

where $\operatorname{clip}(x, a, b)$ squashes $x$ into $[a, b]$ and $\epsilon$ (here the clip range, not the exploration rate of ε-greedy) is typically 0.2.

Read it as two rules:

• The clip removes the incentive to push $\rho_t$ outside $[1-\epsilon, 1+\epsilon]$: beyond the edge the clipped term is flat.
• The min makes sure clipping can only ever make the objective more pessimistic, never more optimistic. You can’t earn credit by moving the ratio the wrong way.

Because $\min(x, y) \le x$, we have $L^{\text{CLIP}} \le L$ sample by sample: the clipped objective is a lower bound on the surrogate, equal to it inside the band. Maximizing a pessimistic estimate is the same spirit as TRPO’s lower bound, at the cost of one min.

The four cases

Each sample falls into one of four cases, depending on the sign of its advantage and on whether its ratio has already moved past the edge in the direction the advantage wants:

Case	Advantage	Ratio	Which term the min picks	Gradient from this sample
1	$\hat A > 0$ (good action)	$\rho \le 1+\epsilon$	unclipped $\rho\hat A$	pushes $\rho$ up
2	$\hat A > 0$	$\rho > 1+\epsilon$	clipped $(1+\epsilon)\hat A$	zero
3	$\hat A < 0$ (bad action)	$\rho \ge 1-\epsilon$	unclipped $\rho\hat A$	pushes $\rho$ down
4	$\hat A < 0$	$\rho < 1-\epsilon$	clipped $(1-\epsilon)\hat A$	zero

So the gradient is exactly zero in two situations: a good action whose probability has already gone up by more than ε, and a bad action whose probability has already gone down by more than ε. Each sample gets to move the policy by about ε in the direction its advantage asks for, and no more.

Notice what is not clipped. A good action with $\rho < 1-\epsilon$ (case 1, at the far left) means the update made a good action less likely, perhaps because other samples dragged it down. The min keeps the unclipped term, so the gradient pulls it back up. Likewise a bad action with $\rho > 1+\epsilon$ (case 3, far right) keeps its gradient and is pushed back down. The clip only switches off the samples that have already got what they wanted.

Written as a gradient, per sample:

$$\nabla_\theta\, \min\big(\rho_t\hat A_t,\, \operatorname{clip}(\rho_t)\hat A_t\big) = \begin{cases} 0 & \text{if } \hat A_t > 0,\ \rho_t > 1+\epsilon \;\text{ or }\; \hat A_t < 0,\ \rho_t < 1-\epsilon,\\[2pt] \rho_t\,\hat A_t\,\nabla_\theta\log\pi_\theta(a_t\mid s_t) & \text{otherwise.} \end{cases}$$

Worked numbers with $\epsilon = 0.2$ (band $[0.8, 1.2]$); these six rows are also the lab’s tests:

$\hat A$	$\rho$	$\rho\hat A$	$\operatorname{clip}(\rho)\hat A$	min	slope $d/d\rho$	case
+2	1.1	2.2	2.2	2.2	+2	1
+2	1.5	3.0	2.4	2.4	0	2
+2	0.5	1.0	1.6	1.0	+2	1 (pulled back up)
−1	0.9	−0.9	−0.9	−0.9	−1	3
−1	0.6	−0.6	−0.8	−0.8	0	4
−1	1.4	−1.4	−1.2	−1.4	−1	3 (pushed back down)

clip range ε =

0.20

ratio ρ = π_θ/π_old =

1.30

PPO-clip objective min(ρÂ, clip(ρ)Â)unclipped ρÂclipped clip(ρ, 1−ε, 1+ε)·Âgradient is zero

What to try: drag across either plot (or use the ρ slider). Watch the green arrow, which shows which way the gradient pushes ρ. It disappears only in the shaded zones. Then shrink ε to 0.05: the band where PPO behaves like vanilla policy gradient becomes very narrow. Finally put ρ at 0.5 and read both panels: the good action is pulled back up, the bad action is left alone.

Clipping is a speed bump, not a wall

Back to the 4-pull bandit batch. With $\epsilon = 0.2$, each sample stops pushing once its ratio crosses the edge. The clipped objective reaches its ceiling of 0.25 as soon as $\rho_A \ge 1.2$, $\rho_B \ge 1.2$ and $\rho_C \le 0.8$; the closest such policy (closest in KL) is $\pi = (0.4, 0.4, 0.2)$, with $\mathrm{KL}(\pi_\text{old}\Vert\pi) = 0.049$. Compare the unclipped surrogate, which ran off to a deterministic policy with infinite KL.

To check this, write the policy as a softmax of three logits, start at uniform, and run plain gradient ascent on $L^{\text{CLIP}}$ until nothing changes any more. The “step size” below is the learning rate of that ascent, on the logits. The result confirms the picture, with a caveat:

step size (on the logits)	final $\pi$ (A, B, C)	$\mathrm{KL}(\pi_\text{old}\Vert\pi)$
0.05	(0.40, 0.40, 0.20)	0.051
1	(0.41, 0.44, 0.16)	0.098
3	(0.43, 0.55, 0.03)	0.576
unclipped, any	(0, 1, 0)	∞

The clip removes the incentive to go further, but it does not stop a big step from jumping past the edge before the gradient switches off. Once past, nothing pulls the ratio back (that sample is now in a flat zone). Many samples also share parameters, so pushing one sample’s ratio moves others. Engstrom et al. (2020) measured this in real PPO runs: ratios can end up well outside $[1-\epsilon, 1+\epsilon]$, and how well PPO keeps its trust region depends heavily on the step size, the number of epochs and other implementation details, not on the clip alone. That is why practical PPO adds a KL check on top (below).

Squeezing one batch: the demo

Here is the whole story in one experiment. Each iteration, two learners collect a batch of $N$ pulls from their own current policy on the 3-armed bandit above (true means A 1.0, B 0.6, C 0.0, reward noise with standard deviation 1), compute advantages as reward minus batch mean, and then take $K$ gradient-ascent steps on the same batch (learning rate 1 on the softmax logits). The only difference: one maximizes the plain surrogate $\hat{\mathbb E}[\rho\hat A]$, the other the PPO-clip objective. Both learners see the same random numbers, so while their policies are identical, their batches are identical too.

epochs per batch K =

40

batch size N =

12

clip ε (PPO only) =

0.20

Run 1 iterationAuto ▶Reset (new seed)

No clip: maximize mean(ρÂ)

PPO-clip

Bars: the current policy π. Dashed line: π_old, the policy that collected this iteration's batch. “avg ” is each arm's mean advantage in the batch.

What to try: with the default seed 1, press “Run 1 iteration” once. The batch happens to make B look best. The unclipped learner jumps to almost always pulling B (KL ≈ 1.7, entropy 0.17); PPO moves a little (KL ≈ 0.04). Press Auto and watch: the unclipped learner never recovers, PPO creeps up to arm A. Then set K = 1: the two curves become identical (why? Exercise 9.7). Raise N to 64 and reset a few times.

Three things to notice:

• Within one batch (the KL chart), the unclipped KL climbs steadily with every epoch: the surrogate keeps rewarding more movement. PPO’s KL rises a little and flattens once most samples are clipped.
• Across batches (the chart of $J$), the unclipped learner often makes a big early gain. Averaged over random seeds, its first iteration takes the expected reward from 0.53 to about 0.83, versus 0.60 for PPO. But in about a third of runs the first batches favour the wrong arm, the policy becomes deterministic, and it is stuck there forever. PPO improves a bit each iteration and, averaged over seeds, overtakes it after six or seven iterations (these figures come from simulating the demo’s default settings over 300 seeds).
• Nothing here is a neural network. The failure is not about function approximation. It is about trusting one noisy batch too much, and about a policy that stops exploring.

PPO-penalty: the adaptive KL version

The PPO paper also proposes a softer variant that is closer to TRPO: subtract a KL penalty instead of clipping.

$$L^{\text{KLPEN}}(\theta) = \hat{\mathbb E}_t\Big[\rho_t(\theta)\hat A_t - \beta\,\mathrm{KL}\big(\pi_\text{old}(\cdot\mid s_t)\,\Vert\,\pi_\theta(\cdot\mid s_t)\big)\Big].$$

The right $\beta$ is hard to know in advance and changes during training, so it is adapted after every update toward a target KL $d_\text{targ}$. With $d$ the measured mean KL after the update:

• if $d < d_\text{targ}/1.5$, set $\beta \leftarrow \beta/2$ (we were too timid);
• if $d > 1.5\,d_\text{targ}$, set $\beta \leftarrow 2\beta$ (we moved too far).

In the paper’s comparison on seven MuJoCo control tasks, clipping with $\epsilon = 0.2$ had the best average normalized score (0.82), the adaptive-KL variants scored 0.68 to 0.74, and the plain surrogate with neither clipping nor penalty scored −0.39: on average worse than a random policy, because on one task (HalfCheetah) it ended up far worse than random. That comparison, plus its simplicity, is why “PPO” today almost always means PPO-clip.

The full PPO algorithm

PPO in practice is an actor-critic (Chapter 8) with the clipped objective as its actor loss. One combined loss trains everything:

$$\mathcal L(\theta, \phi) = -L^{\text{CLIP}}(\theta) + c_1\,\hat{\mathbb E}_t\big[(V_\phi(s_t) - R_t)^2\big] - c_2\,\hat{\mathbb E}_t\big[\mathcal H\big(\pi_\theta(\cdot\mid s_t)\big)\big],$$

where $R_t = \hat A_t + V_\text{old}(s_t)$ is the critic’s target, $\mathcal H$ is the entropy (a bonus that slows the policy’s slide into determinism), and $c_1, c_2$ are weights (the paper’s Atari runs used $c_1 = 1$, $c_2 = 0.01$). The minus signs are there because we minimize $\mathcal L$.

PPO-clip, actor-critic style
for iteration = 1, 2, ...:
  1. Rollout: run π_θ for T steps (often in several parallel environments).
     Store s_t, a_t, r_t, done_t, and, frozen at sampling time:
         logp_old_t = log π_θ(a_t | s_t),   V_old_t = V_φ(s_t)
  2. Advantages: Â_t = GAE(γ, λ) from r, done, V_old.   Targets: R_t = Â_t + V_old_t
  3. for epoch = 1..K:
       shuffle the T samples; for each minibatch M:
         Â ← (Â − mean_M(Â)) / std_M(Â)                       (advantage normalization)
         ρ = exp(log π_θ(a|s) − logp_old)
         L_clip = mean_M min(ρ·Â, clip(ρ, 1−ε, 1+ε)·Â)
         L_V    = mean_M (V_φ(s) − R)²
         H      = mean_M entropy(π_θ(·|s))
         loss   = −L_clip + c1·L_V − c2·H
         one Adam step on loss, after clipping the global gradient norm
       if approx_kl(full batch) > target_kl: break           (early stopping)

Two details in this pseudocode are easy to get wrong. First, logp_old and V_old are recorded once, when the data is collected, and never recomputed during the epochs: they define the trust region’s centre. Second, only the minibatch’s current log-probabilities carry gradients.

Typical settings:

Hyperparameter	PPO paper (MuJoCo)	This chapter’s lab (CartPole)
rollout length $T$	2048	512
epochs $K$	10	10
minibatch size	64	64
Adam learning rate	$3\times10^{-4}$	$10^{-3}$, annealed linearly to 0
$\gamma$, $\lambda$	0.99, 0.95	0.99, 0.95
clip $\epsilon$	0.2 (best of 0.1, 0.2, 0.3)	0.2
value-loss weight $c_1$	not needed (separate policy and value networks)	0.5, on $\tfrac12(V_\phi - R)^2$
entropy weight $c_2$	0	0

The implementation details that matter

PPO’s paper is short; working implementations contain many small choices that are not in it. Engstrom et al. (2020) found that some of these “code-level optimizations” matter as much as the clip itself, and Huang et al. (2022) catalogued 37 of them. These are the ones worth understanding:

• Advantage normalization. Rescale $\hat A$ to mean 0, standard deviation 1 within each minibatch. The clip limits the ratio, but the size of each step also scales with $\lvert\hat A\rvert$. Normalizing makes the learning rate and $\epsilon$ mean the same thing whether rewards are 0.01 or 1000. (CartPole’s +1-per-step rewards give raw advantages of order 10 to 100.)
• Value clipping. The critic gets its own clip: $L_V = \max\big((V - R)^2, (V_\text{clip} - R)^2\big)$ with $V_\text{clip} = V_\text{old} + \operatorname{clip}(V - V_\text{old}, -\epsilon, \epsilon)$. It appears in OpenAI’s original code, but large studies found no evidence that it helps (Engstrom et al., 2020) and some evidence that it hurts (Andrychowicz et al., 2021). The lab implements it as an option, off by default.
• Gradient-norm clipping. Rescale the whole gradient if its norm exceeds a threshold (0.5 is common). A cheap guard against one bad minibatch.
• Learning-rate annealing. Decay the learning rate linearly to 0 over training, so late updates are gentle.
• Approx-KL early stopping. After each epoch, estimate $\mathrm{KL}(\pi_\text{old}\Vert\pi_\theta)$ on the batch; stop the epochs if it exceeds a target such as 0.01–0.05 (OpenAI’s Spinning Up implementation uses 0.01 by default). This turns the clip’s soft speed bump into something much closer to TRPO’s hard constraint.
• Orthogonal initialization with a small final policy layer (so the initial policy is near-uniform), and observation normalization for tasks whose state variables have very different scales.

The two numbers to watch

You can’t compute the exact KL between two neural policies over all states, but you can estimate it from the batch, using the stored $\log\pi_\text{old}(a_t\mid s_t)$ and the current $\log\pi_\theta(a_t\mid s_t)$. Since $a_t \sim \pi_\text{old}$:

$$\mathrm{KL}(\pi_\text{old}\Vert\pi_\theta) = \mathbb E_{a\sim\pi_\text{old}}[-\log\rho] \approx \hat{\mathbb E}_t\big[(\rho_t - 1) - \log\rho_t\big].$$

Both forms are unbiased, because $\mathbb E_{a\sim\pi_\text{old}}[\rho] = 1$, so adding $\rho - 1$ adds zero on average. The second form, called $k_3$ in Chapter 11, has the advantage that every single sample is $\ge 0$ (because $\log x \le x - 1$), and its variance is usually lower. This is the approx KL that PPO implementations log.

The clip fraction is the fraction of samples with $\lvert\rho_t - 1\rvert > \epsilon$: how many samples have run into the edge of the band.

What you see	What it usually means
approx KL ≈ 0, clip fraction ≈ 0	The policy is barely moving: learning rate too small, or advantages near zero
moderate KL, clip fraction around 0.05–0.2	A typical healthy run (our CartPole lab: KL 0.001–0.01 per update)
KL jumps to 0.1 or more, clip fraction above 0.3	Updates too aggressive: lower the learning rate or K, or set a target KL
entropy falls to near 0 early	Premature collapse: the policy has stopped exploring

These ranges are rules of thumb that vary with the problem; what matters most is a change in the numbers. In the lab’s run with no clipping and aggressive settings, the approx KL per update is mostly between 1 and 10, and the entropy collapses after the first update.

Summary

• Vanilla policy-gradient steps are fragile because parameter distance is not behaviour distance, and because a bad step corrupts the data the policy collects next. They are wasteful because each batch is used once.
• The surrogate $L(\theta) = \hat{\mathbb E}_t[\rho_t(\theta)\hat A_t]$ uses importance ratios to evaluate a new policy on old data. It matches the true objective in value and gradient at $\theta_\text{old}$, and becomes unreliable as $\pi_\theta$ moves away (noisy weights, and states the old policy rarely visited).
• Trust regions limit each update by $\mathrm{KL}(\pi_\text{old}\Vert\pi_\theta) \le \delta$. To second order KL is $\tfrac12\Delta\theta^\top F\Delta\theta$ with $F$ the Fisher information, and the solution is the natural gradient $F^{-1}g$. TRPO computes it with conjugate gradient and a line search.
• PPO-clip maximizes $\hat{\mathbb E}_t[\min(\rho_t\hat A_t, \operatorname{clip}(\rho_t, 1\pm\epsilon)\hat A_t)]$, a pessimistic lower bound on the surrogate. A sample’s gradient is exactly zero when a good action’s ratio exceeds $1+\epsilon$ or a bad action’s ratio falls below $1-\epsilon$; otherwise it is the importance-weighted policy gradient.
• The clip removes the incentive to move further but is not a hard wall; big steps and many epochs can still overshoot. Watch approx KL and clip fraction, and stop early when the KL gets too large.
• Full PPO = rollouts → GAE → K epochs of minibatch updates on clipped policy loss + value loss − entropy bonus, with advantage normalization, gradient clipping and learning-rate annealing.
• The trust-region KL to $\pi_\text{old}$ is not the RLHF leash to $\pi_\text{ref}$; LLM training uses both.

Exercises

Exercise 9.1Why not just reuse the batch with the REINFORCE loss?Concept●○○

A friend proposes a shortcut: “Collect a batch, then run 10 epochs of Adam on the usual actor loss $-\hat{\mathbb E}_t[\log\pi_\theta(a_t\mid s_t)\hat A_t]$. No ratios needed.” What goes wrong after the first step? Make it concrete for a bandit whose batch advantages sum to zero (as they do when the baseline is the batch mean).

Solution

The gradient $\hat{\mathbb E}_t[\nabla\log\pi_\theta(a_t\mid s_t)\hat A_t]$ estimates the policy gradient only when the $a_t$ are sampled from $\pi_\theta$ itself. After one step the samples come from $\pi_\text{old} \ne \pi_\theta$, and the correct gradient of the improvement is $\hat{\mathbb E}_t[\rho_t\nabla\log\pi_\theta\,\hat A_t]$, which is $\nabla L$. Without the ratios, you are maximizing $\hat{\mathbb E}_t[\log\pi_\theta(a_t)\hat A_t]$, an advantage-weighted log-likelihood of a fixed dataset, which has nothing to do with expected reward once you leave $\theta_\text{old}$.

Concretely, for a softmax bandit with logits $z$, $\nabla_z\log\pi(a) = e_a - \pi$. So

$$\nabla_z \frac1N\sum_i \log\pi(a_i)\hat A_i = \frac1N\sum_i \hat A_i e_{a_i} - \pi\,\frac1N\sum_i\hat A_i = \frac1N\sum_i \hat A_i e_{a_i},$$

using $\sum_i \hat A_i = 0$. This gradient is a constant vector (the summed advantage of each arm) that never shrinks. Every epoch adds the same amount to each logit, forever: the arm with the largest summed advantage in this one batch takes over and the policy goes deterministic. With ratios, the unclipped surrogate also runs away (the text’s 4-pull example), which is why we also need the trust region. But at least it is running toward the right objective.

I've done this one

Exercise 9.2Surrogate and true objective agree to first orderPen & paper●●○

This exercise asks you to redo the chapter’s derivation yourself, without looking. Using the performance difference lemma, show that $L(\theta_\text{old}) = 0$, the same value as $J(\theta) - J(\theta_\text{old})$ at $\theta = \theta_\text{old}$, and that $\nabla_\theta L(\theta_\text{old}) = \nabla_\theta J(\theta_\text{old})$, where $L(\theta) = \mathbb E_{\tau\sim\pi_\text{old}}[\sum_t\gamma^t\rho_t(\theta)A^{\pi_\text{old}}(s_t,a_t)]$. Then show that $\nabla_\theta L(\theta_\text{old})$ is the policy gradient theorem’s expression from Chapter 7.

Hint

Write both $J(\theta) - J(\theta_\text{old})$ and $L(\theta)$ as $\sum_t\gamma^t\sum_s p_t(s)\,f_\theta(s)$, with different state distributions $p_t$. What is $f_{\theta_\text{old}}(s)$? Then use the product rule.

Solution

Let $f_\theta(s) = \sum_a\pi_\theta(a\mid s)A^{\pi_\text{old}}(s,a)$. By the performance difference lemma, $J(\theta) - J(\theta_\text{old}) = \sum_t\gamma^t\sum_s p_t^\theta(s)f_\theta(s)$, where $p_t^\theta$ is the state distribution at time $t$ under $\pi_\theta$. Importance-weighting the actions shows $L(\theta) = \sum_t\gamma^t\sum_s p_t^{\theta_\text{old}}(s)f_\theta(s)$: same $f$, old state distribution.

Values. $f_{\theta_\text{old}}(s) = \sum_a\pi_\text{old}(a\mid s)\big(Q^{\pi_\text{old}}(s,a) - V^{\pi_\text{old}}(s)\big) = V^{\pi_\text{old}}(s) - V^{\pi_\text{old}}(s) = 0$ for every $s$. So both expressions are 0 at $\theta_\text{old}$.

Gradients. By the product rule,

$$\nabla_\theta[J(\theta) - J(\theta_\text{old})] = \sum_t\gamma^t\sum_s\big[\nabla_\theta p_t^\theta(s)\,f_\theta(s) + p_t^\theta(s)\,\nabla_\theta f_\theta(s)\big].$$

At $\theta_\text{old}$ the first term is zero because $f_{\theta_\text{old}} \equiv 0$, and the second is $\sum_t\gamma^t\sum_s p_t^{\theta_\text{old}}(s)\nabla f_\theta(s) = \nabla L(\theta_\text{old})$. Since $J(\theta_\text{old})$ is a constant, $\nabla J(\theta_\text{old}) = \nabla L(\theta_\text{old})$.

Policy gradient form. $\nabla_\theta\rho_t = \rho_t\nabla_\theta\log\pi_\theta(a_t\mid s_t)$ and $\rho_t = 1$ at $\theta_\text{old}$, so

$$\nabla L(\theta_\text{old}) = \mathbb E_{\tau\sim\pi_\text{old}}\Big[\sum_t\gamma^t\,\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,A^{\pi_\text{old}}(s_t,a_t)\Big],$$

the policy gradient theorem with the advantage as the weight (Chapters 7–8). The two objectives differ only at second order, through $\nabla p_t^\theta$, the change in which states get visited.

I've done this one

Exercise 9.3Work through the four clip casesPen & paper●○○

Take $\epsilon = 0.1$. For each sample, compute the PPO-clip term $\min(\rho\hat A, \operatorname{clip}(\rho, 0.9, 1.1)\hat A)$, its slope with respect to $\rho$, and its case number (1–4) from the table in the text.

(a) $\hat A = +1, \rho = 1.05$ (b) $\hat A = +1, \rho = 1.25$ (c) $\hat A = +1, \rho = 0.7$ (d) $\hat A = -2, \rho = 1.3$ (e) $\hat A = -2, \rho = 0.85$ (f) $\hat A = -2, \rho = 0.95$

(g) Show that for any $\rho$ and $\hat A$, the PPO-clip term is at most $\rho\hat A$.

Solution

	$\rho\hat A$	$\operatorname{clip}(\rho)\hat A$	min	slope	case
(a)	1.05	1.05	1.05	+1	1 (inside the band)
(b)	1.25	1.10	1.10	0	2 (good action, already boosted)
(c)	0.70	0.90	0.70	+1	1 (went the wrong way; pulled back up)
(d)	−2.60	−2.20	−2.60	−2	3 (went the wrong way; pushed back down)
(e)	−1.70	−1.80	−1.80	0	4 (bad action, already suppressed)
(f)	−1.90	−1.90	−1.90	−2	3 (inside the band)

The slope is $\hat A$ when the min picks the unclipped term (and $\rho$ is free), and 0 when it picks the clipped term outside the band, where $\operatorname{clip}(\rho)$ is constant.

(g) $\min(x, y) \le x$ for any numbers, so $\min(\rho\hat A, \operatorname{clip}(\rho)\hat A) \le \rho\hat A$. Averaging over samples, $L^{\text{CLIP}} \le L$: the clipped objective is a pessimistic lower bound on the surrogate, with equality whenever every ratio is inside the band.

I've done this one

Exercise 9.4KL between two categorical policies, by handPen & paper●●○

In some state, $\pi_\text{old} = (0.5, 0.3, 0.2)$ and $\pi_\text{new} = (0.4, 0.4, 0.2)$ over three actions.

(a) Compute $\mathrm{KL}(\pi_\text{old}\Vert\pi_\text{new})$ and $\mathrm{KL}(\pi_\text{new}\Vert\pi_\text{old})$.

(b) Compute the second-order approximation $\tfrac12\sum_a(\Delta\pi_a)^2/\pi_\text{old}(a)$ (the Fisher form written in terms of the probabilities). How close is it?

(c) For each action, compute the per-sample estimators $k_1 = -\log\rho$ and $k_3 = (\rho - 1) - \log\rho$, with $\rho = \pi_\text{new}/\pi_\text{old}$. Check that both average (under $\pi_\text{old}$) to the KL from (a). Which estimator can be negative for a single sample?

Hint

Natural logs. For (c), the ratios are 0.8, 4/3 and 1. To average “under π_old”, weight each action’s value by 0.5, 0.3, 0.2.

Solution

(a) $\mathrm{KL}(\pi_\text{old}\Vert\pi_\text{new}) = 0.5\ln\frac{0.5}{0.4} + 0.3\ln\frac{0.3}{0.4} + 0.2\ln 1 = 0.5(0.22314) + 0.3(-0.28768) + 0 = 0.11157 - 0.08630 = 0.02527$ nats.

$\mathrm{KL}(\pi_\text{new}\Vert\pi_\text{old}) = 0.4\ln\frac{0.4}{0.5} + 0.4\ln\frac{0.4}{0.3} = 0.4(-0.22314) + 0.4(0.28768) = 0.02582$ nats.

They differ by about 2%: KL is asymmetric, but for small changes the two directions agree to second order.

(b) $\Delta\pi = (-0.1, +0.1, 0)$: $\tfrac12\big(\frac{0.01}{0.5} + \frac{0.01}{0.3} + 0\big) = \tfrac12(0.02 + 0.03333) = 0.02667$. About 5% above the exact value; the difference is the third-order term.

(c)

action	$\pi_\text{old}$	$\rho$	$k_1 = -\ln\rho$	$k_3 = \rho - 1 - \ln\rho$
1	0.5	0.8	+0.2231	0.0231
2	0.3	1.3333	−0.2877	0.0457
3	0.2	1	0	0

Averages: $k_1$: $0.5(0.2231) + 0.3(-0.2877) = 0.1116 - 0.0863 = 0.0253$. $k_3$: $0.5(0.0231) + 0.3(0.0457) = 0.0116 + 0.0137 = 0.0253$. Both match the KL (up to rounding). $k_1$ is negative for action 2, so with few samples it can give a negative “KL”. $k_3$ is never negative because $\ln x \le x - 1$; its per-sample values (0.023, 0.046, 0) are also much less spread out than $k_1$‘s (0.22, −0.29, 0). That is why PPO code logs $k_3$ as “approx KL”.

I've done this one

Exercise 9.5The natural gradient of a two-action policyPen & paper●●●

Let $\pi_\theta(\text{right}) = \sigma(\theta)$, with “right” paying reward 1 and “left” paying 0.

(a) Show that $F(\theta) = \sigma(1-\sigma)$ and that the plain gradient is $g = \sigma(1-\sigma)$.

(b) Solve $\max_\Delta g\Delta$ subject to $\tfrac12F\Delta^2 \le \delta$, and evaluate the step for $\delta = 0.01$ at $\theta = 0$ and $\theta = 4$.

(c) Starting from $\theta = 0$, how many plain gradient steps with learning rate $\alpha = 1$, versus natural-gradient steps with $\delta = 0.01$, do you need to reach $\pi(\text{right}) \ge 0.99$? (Estimate; a short loop in Python is fine.) What does this say about plain gradients near deterministic policies?

Hint

For (a), $F = \mathbb E_{a\sim\pi}[(\partial_\theta\log\pi(a))^2]$ with $\partial_\theta\log\sigma = 1-\sigma$ and $\partial_\theta\log(1-\sigma) = -\sigma$. For (b), in one dimension the constraint just says $\lvert\Delta\rvert \le \sqrt{2\delta/F}$.

Solution

(a) $F = \sigma(1-\sigma)^2 + (1-\sigma)\sigma^2 = \sigma(1-\sigma)\big[(1-\sigma) + \sigma\big] = \sigma(1-\sigma)$. And $J = \sigma(\theta)$, so $g = \sigma'(\theta) = \sigma(1-\sigma)$.

(b) $g > 0$, so we want the largest $\Delta$ allowed: $\Delta = \sqrt{2\delta/F}$. (The general formula gives the same: $F^{-1}g = 1$ and $g^\top F^{-1}g = F$.) At $\theta = 0$: $F = 0.25$, $\Delta = \sqrt{0.08} = 0.283$. At $\theta = 4$: $F = 0.0177$, $\Delta = \sqrt{0.02/0.0177} = 1.064$.

(c) You need $\theta \ge \ln 99 \approx 4.6$. Natural steps: the KL per step stays near 0.01 and the steps grow (0.28 at first, about 1.0 near $\theta=4$), so it takes 11 steps. Plain steps add $\sigma(1-\sigma)$ each time: 0.25 per step at first, but below 0.02 per step once $\theta > 4$, so it takes about 100 steps (a quick loop gives 107). The plain gradient vanishes exactly when the policy becomes confident, so it is slowest precisely when you’d like to finish the job, and it would also be slowest to undo a confident mistake. The natural gradient rescales by $F^{-1}$ and moves at a constant speed in behaviour space.

I've done this one

Exercise 9.6Why clip, and not just a KL penalty?Concept●●○

PPO-penalty with an adaptive $\beta$ is closer to TRPO’s theory than PPO-clip is. Give three reasons why practitioners still prefer the clip. Then explain why LLM training uses both a clip and a KL penalty.

Hint

Think about what $\beta$ needs to be at the start versus the end of training, about per-sample versus batch-average control, and about what each one is measured against.

Solution

1. Tuning. The right $\beta$ depends on the scale of the advantages and on the stage of training, so a fixed $\beta$ is wrong most of the time. The adaptive rule fixes this only one update late, by doubling or halving after it measured the KL. $\epsilon = 0.2$ works across a wide range of problems, especially with normalized advantages.
2. Local control. The penalty acts on the batch-average KL: a few samples can still move a lot while the average stays small. The clip acts per sample: each sample stops pushing as soon as it has moved its own probability by about ε.
3. Evidence and simplicity. In the PPO paper’s MuJoCo comparison, clip with $\epsilon = 0.2$ scored 0.82 against 0.68–0.74 for adaptive KL. The clip also needs only the log-probability of the action taken, while an exact KL penalty needs the whole action distribution at every state (or an estimator).

In LLM training the two do different jobs. The clip is a trust region around $\pi_\text{old}$, the policy that generated the current batch: it keeps one update small. The KL penalty is to $\pi_\text{ref}$, the frozen SFT model: a leash that keeps the policy close to fluent, reasonable language for the whole of training, so that it doesn’t drift into regions where the reward model can be fooled (Chapters 11, 12 and 19). Replacing one with the other would not work: a trust region resets every iteration, so a policy can drift arbitrarily far from $\pi_\text{ref}$ in many small steps.

I've done this one

Exercise 9.7Experiments with the K-epoch demoExplore the demo●●○

Use the demo “K epochs on one batch, with and without the clip”.

(a) Set K = 1 and run several iterations. Why are the two learners’ curves identical?

(b) With the default settings and seed 1, run Auto to the end. The unclipped learner’s entropy goes to 0 and its expected reward is stuck at 0.6. Explain, in terms of what its batches look like, why it can never recover.

(c) Set N = 64 and press Reset several times, running a few iterations each time. Does the unclipped learner still sometimes lock onto the wrong arm? What does it still do every time?

(d) With N = 12 and K = 100, set ε to 0.05 and then 0.5. Describe how PPO’s KL after one iteration and its speed of improvement change.

Hint

For (a), ask what every ratio equals before the first step, and which of the four clip cases that is. For (b), suppose every pull in the batch is arm B: what do the advantages sum to, and what does each logit’s gradient become? For (d), read the KL value in the PPO card after one iteration, then press Reset and repeat a few times.

Solution

(a) With K = 1 each learner takes a single step from $\theta_\text{old}$, where every $\rho = 1$, inside the band. No sample is clipped, so the two objectives have identical gradients (both equal the vanilla policy gradient). The two learners start with the same policy and use the same random numbers, so they collect identical batches, take identical steps, and stay identical forever. PPO with one epoch is vanilla policy gradient.

(b) After a few iterations the policy pulls (say) B with probability about 1, so a batch is N pulls of B. The advantages are reward minus batch mean, so they sum to zero over the batch, and all samples have the same ratio. The gradient on the logits, $\frac1N\sum_i\rho\hat A_i(e_B - \pi)$, is therefore zero. There is no data about A or C at all, so nothing can tell the learner that A is better: the gradient is zero and the evidence is missing. This is the policy collapse from the start of the chapter.

(c) Locking onto the wrong arm becomes rarer, because with 64 samples B rarely looks better than A by chance (in a 300-seed simulation, about 8% of runs are on the wrong arm after three iterations). But the unclipped learner still jumps a long way on its very first batch: its KL after one iteration is about 1.6 in a typical run, and above 0.9 in 9 runs out of 10. After two iterations its entropy is typically about 0.1 (from 1.10 at the start), so it is nearly deterministic. With a 3-armed bandit and a big batch that happens to work, because it usually commits to the right arm. In a large problem, committing fully on the basis of one batch means losing exploration everywhere.

(d) With ε = 0.05 each iteration moves only a little (KL around 0.01), so improvement is slow but very steady. With ε = 0.5 the band is wide: a single iteration moves the KL to around 0.2–0.3, so progress is faster, but the learner over-commits to a lucky arm more often, a milder version of the unclipped learner. (Simulating 300 seeds: after 10 iterations, ε = 0.5 ends on the wrong arm about 10% of the time, versus about 6% for ε = 0.2. On this easy bandit the faster ε = 0.5 still has the higher average reward; on hard problems the extra risk usually costs more.) ε trades speed for safety, which is why it is a hyperparameter, with 0.1–0.3 the usual range.

I've done this one

Exercise 9.8What goes wrong with too many epochs?Concept●●○

PPO is clipped, so why not run K = 100 epochs on every batch to squeeze out every drop of data? List what happens as K grows, and say which diagnostic shows each effect.

Hint

Look at the step-size table in “Clipping is a speed bump, not a wall”, and at the “PPO, aggressive” curve in the lab’s plot.

Solution

• More samples hit the edge. The clip fraction rises; the remaining gradient comes from fewer and fewer samples, mostly ones that went “the wrong way” (the pull-back cases), so later epochs are increasingly noisy and add little.
• The ratio overshoots. The clip is not a wall: a step can carry a ratio past the edge, and once there nothing pulls it back, while gradients from other samples sharing the parameters keep moving it. In the text’s 4-pull example the clipped objective, optimized with a large step size, ended at KL 0.58 instead of 0.05. The approx KL grows with K.
• Overfitting to the batch. The advantages are noisy estimates. Many passes fit that noise, in the same way that optimizing the unclipped surrogate to convergence picked the lucky arm.
• The critic overfits too, to this batch’s returns, which makes the next batch’s advantages worse. The value loss on fresh data goes up.
• Entropy drops faster, and exploration with it.

The fix is not to pick K perfectly but to stop when the policy has moved enough: approx-KL early stopping (Task 4 of the lab) ends the epochs once the KL to $\pi_\text{old}$ exceeds a target.

I've done this one

Exercise 9.9Lab, Tasks 1–3: the PPO losses and diagnosticsCode●●○

In exercises/09-ppo/ppo.py, implement clipped_surrogate_loss (Task 1), approx_kl and clip_fraction (Task 2), and value_loss (Task 3). Run python -m pytest -q -k "task1 or task2 or task3". The Task 1 tests check the six worked cases from the text, value and gradient. If one fails, the message tells you which case.

Hint

Compute the ratio as torch.exp(logp_new - logp_old) rather than dividing probabilities (stable with tiny probabilities). torch.min of two tensors is element-wise. Don’t forget the minus sign: optimizers minimize.

Solution

The heart of the reference solution (solutions/ppo.py):

ratio = torch.exp(logp_new - logp_old)
unclipped = ratio * adv
clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * adv
loss = -torch.min(unclipped, clipped).mean()

torch.clamp has zero gradient outside its range, and torch.min routes the gradient to whichever term it picked. Autograd therefore produces exactly the four-case gradient from the text, with no if statements.

For the diagnostics, with log_ratio = logp_new - logp_old: approx_kl = ((exp(log_ratio) - 1) - log_ratio).mean().item() and clip_fraction = ((exp(log_ratio) - 1).abs() > clip_eps).float().mean().item(), both under torch.no_grad().

The value loss with clipping: v_clip = values_old + torch.clamp(values - values_old, -eps, eps), then 0.5 * torch.max((values - returns)**2, (v_clip - returns)**2).mean(). Note the max: as with the policy, the clipped version is the pessimistic one (the larger loss).

I've done this one

Exercise 9.10Lab, Task 4: full PPO on CartPoleCode●●●

Implement ppo_update (Task 4): K epochs of shuffled minibatches, advantage normalization, the combined loss, gradient-norm clipping and approx-KL early stopping. When all tests pass, run python run.py (about 2 minutes). It trains four variants: PPO, PPO with aggressive settings (K = 40, learning rate 3e-3), the same aggressive settings with no clip, and one epoch per batch. Explain each curve in the plot. Then set target_kl=0.01 for the aggressive variants: how many of the 40 epochs actually run?

Hint

Follow the numbered steps in the docstring. Use the stored logp_old for the ratio; never recompute it during the epochs. The early-stopping check uses the full batch, after each epoch.

Solution

The reference loop (solutions/ppo.py) follows the docstring step by step: for each epoch, torch.randperm the indices; for each minibatch, compute dist = agent.dist(obs), logp_new, the entropy, normalize the minibatch advantages, form loss = pl + vf_coef * vl - ent_coef * entropy, then zero_grad, backward, clip_grad_norm_, step. After the epoch, if target_kl is set, recompute log-probs on the whole batch under torch.no_grad() and break if approx_kl exceeds it.

What the plot shows (reference solution, seed 0, 40k steps):

• PPO (K = 10) reaches the 500-step ceiling in roughly 20–30k steps. Approx KL per update stays around 0.001–0.01.
• PPO, aggressive learns about as fast. Its clip fraction is higher (often 0.1–0.3) and its KL a few times larger, but still around 0.01: the clip absorbs most of the extra aggressiveness. It is a little less steady late in training (in the reference run it dips from 500 to about 390 after 30k steps and then recovers part of the way).
• No clip, aggressive: approx KL of 1–10 per update, a hundred to a thousand times larger. The policy’s entropy collapses to almost 0 after the first update, and the return is erratic and crashes (in the reference run it falls from about 390 to near 20 around step 20k). This is the chapter’s opening story, live.
• One epoch is stable but slow (about 300 at 40k steps): each batch is used for a single pass.

With target_kl=0.01 (reference solution, seed 0), the clipped aggressive variant runs a median of about 12 of its 40 epochs, and the unclipped one a median of only 2. The unclipped variant then no longer collapses: its entropy stays above 0.3 and it reaches a return of about 400 by 40k steps. Early stopping alone already provides a crude trust region. The exact counts depend on the seed.

I've done this one

Further reading

• Schulman et al. (2017), Proximal Policy Optimization Algorithms. Short and readable. Section 3 is the clipped objective (with the four-case figure), section 4 the adaptive KL, Table 1 the comparison quoted above.
• Huang et al. (2022), The 37 Implementation Details of Proximal Policy Optimization. The best practical guide: every trick in real PPO code, with a reference implementation. Read it before you write PPO for anything serious.
• Schulman et al. (2015), Trust Region Policy Optimization. The theory: the monotonic-improvement bound, the KL constraint, conjugate gradient and the line search (Appendix C).
• OpenAI Spinning Up, PPO and TRPO. Clear, compact explanations with pseudocode and clean implementations.
• Engstrom et al. (2020), Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. Shows that code-level details drive much of PPO’s performance, and that the clip alone does not keep ratios in the band.
• Andrychowicz et al. (2021), What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study. Hundreds of thousands of training runs that measure which PPO choices matter; a source of good defaults.
• Schulman (2020), Approximating KL Divergence. A short note on the $k_1$, $k_2$, $k_3$ estimators behind “approx KL”; Chapter 11 returns to it.

Next, Chapter 10 steps back from the policy-gradient line and maps the rest of RL (planning and AlphaZero, imitation, offline and maximum-entropy RL), each idea tied to where it reappears with LLMs.