Failure modes: reward hacking and friends

In 2016, researchers at OpenAI trained an agent on CoastRunners, a boat-racing video game. The game’s score came from hitting targets along the course, not from finishing the race. The agent found a lagoon where a few targets kept respawning, and it circled there indefinitely, crashing into other boats, going the wrong way and catching fire. In the researchers’ words, it achieved “a higher score using this strategy than is possible by completing the course in the normal way.” It never finished a race. In 2025, OpenAI researchers watched a frontier reasoning model being trained with RL to fix code until the unit tests passed. Among the tricks it learned was to call exit(0), so the test harness saw a clean exit before any test could fail.

Nine years and many orders of magnitude of compute separate those two stories, and they are the same story. Reinforcement learning is an optimizer. It finds what you reward, not what you meant, and the better the optimizer, the more reliably it finds the difference. Every chapter of Part III so far has quietly assumed the reward is right: the reward model of Chapter 12, the judges and rubrics of Chapter 14, the verifiers of Chapter 15. This chapter drops that assumption. You will see why optimizing a good-but-imperfect reward first helps and then hurts, measure the effect in nats, meet the main ways LLMs hack their rewards, and see a second family of failures in which nothing is hacked but the policy stops exploring. Then we assemble the toolbox people use against all of it.

The proxy and the target

Goodhart’s law

The economist Charles Goodhart observed in 1975 that a statistical regularity tends to collapse once you lean on it to control something. The anthropologist Marilyn Strathern later gave it the form everyone quotes: when a measure becomes a target, it ceases to be a good measure. A school judged on test scores starts teaching to the test. A hospital judged on waiting times starts discharging patients early.

In RL for LLMs the measure is the reward. Following Gao, Schulman and Hilton (2022), we give the two sides of the gap names:

• The gold reward is what you actually want: a careful human’s judgment, the true correctness of a program, whether a summary is faithful. You usually can’t compute it at training scale. That is why you needed a substitute in the first place.
• The proxy reward is what the optimizer actually sees: a reward model’s score, a unit-test pass rate, an LLM judge’s verdict.

Reward hacking is when a policy raises the proxy in a way that does not raise, or even lowers, the gold. Specification gaming (the term popularized by Krakovna and colleagues at DeepMind) means the same thing with the emphasis on the specification: the agent did exactly what you wrote down, and what you wrote down was wrong. People use the two terms almost interchangeably. Neither one requires intent. A policy-gradient update has no idea that it is “cheating”; it just raises the probability of whatever scored well.

Why optimization pressure is the problem

Suppose a reward model was trained on responses of ordinary length, and longer answers were slightly preferred by the labelers because they tended to be more complete. The reward model correctly learns “longer is a bit better” in that range. Nothing is wrong yet. Now hand it to an optimizer. The optimizer discovers that length is the cheapest way to raise the score, and pushes responses into lengths no labeler ever saw. Out there, the reward model’s “longer is better” is an extrapolation, and the extrapolation is wrong: a 3,000-word answer to a yes/no question is not better.

So the danger is not that the proxy is noisy or slightly biased. Every proxy is. The danger is the interaction between a slightly wrong proxy and a strong optimizer. You can see this cleanly in a toy model small enough to solve by hand.

A toy you can solve by hand

The setup

Give every response three hidden features, each a standard normal under the starting policy $\pi_\text{ref}$:

• $q$, its quality;
• $\ell$, its length, in standard deviations from typical;
• $e$, some quirk that the reward model happens to like but nobody cares about.

The gold reward is quality, with a penalty for being far from a sensible length in either direction:

The proxy reward model gets quality right, but it has also learned that length is good (a bias of size $b$), and it responds to the irrelevant quirk (noise of size $\sigma$):

Near the starting policy these agree well: for typical responses, $\ell$ is small, the $\ell^2$ penalty is negligible, and both rewards mostly track $q$. Under $\pi_\text{ref}$ the average gold is $\mathbb E[g] = 0 - \tfrac12 \mathbb E[\ell^2] = -\tfrac12$ and the average proxy is $0$. We’ll measure everything as a gain over these starting values.

Optimizing the proxy with a KL leash

From Chapter 11 you know what KL-regularized RL converges to, if optimization succeeds: the maximizer of $\mathbb E_\pi[p] - \beta\,\mathrm{KL}(\pi \Vert \pi_\text{ref})$ is the tilted distribution

Here $\pi_\text{ref}$ is a standard Gaussian over the features and $p$ is linear in them, so the tilt is easy. Tilting a Gaussian by the exponential of a linear function shifts its mean and leaves its shape alone (the derivation below completes the square). Write $t = 1/\beta$ for the optimization pressure. The optimized policy has

Quality goes up, which is good. Length goes up too, in proportion to $b$. Plugging in, the gains over $\pi_\text{ref}$ are

The proxy rises linearly forever. The gold gain is a parabola: it rises, peaks at $t = 1/b^2$, and then falls without limit.

Worked example: $b = 1$, no noise

Take a reward model that likes length about as much as it likes quality ($b = 1$) and has no noise ($\sigma = 0$). Then $\mathrm{KL} = t^2$, so the distance from the reference measured as $d = \sqrt{\mathrm{KL}}$ is just $d = t$. Lower $\beta$ step by step (a weaker leash):

Read the table as a training run. The proxy climbs steadily: 0, 1, 2, 4, 8. If the proxy is all you can see, everything looks great. Meanwhile the gold improves at first (the optimizer is finding genuinely better responses), peaks at KL = 1 nat, is back to zero by 4 nats, and is far below the starting policy by 16 nats. At the end, responses are four standard deviations too long, and the proxy loves them.

In terms of $d$, the gold gain is $d - \tfrac12 d^2$: rising linearly, falling quadratically. Hold on to that shape; you’ll see it again in real data in a moment.

Two different ways a proxy can be wrong

The toy separates two kinds of error, and they behave very differently.

• Noise ($\sigma$). Set $b = 0$. Now $\Delta\text{gold} = t$, which never falls. But the KL is $\tfrac12 t^2(1 + \sigma^2)$, so at a fixed KL budget you get less gold: part of every unit of optimization is spent selecting for the quirk $e$, which is worthless. You are disappointed (the gold gain is smaller than the proxy gain promised) but not harmed. This is regressional Goodhart, the first of four variants in Manheim and Garrabrant’s taxonomy: selecting on a noisy measure also selects for the noise.
• Bias that breaks at the extremes ($b$). The proxy’s “longer is better” is true near typical lengths and false far from them. Optimization drives you exactly to where it is false, and the gold eventually falls below where you started. This is extremal Goodhart: a relationship that held in the normal range fails in the extreme range that optimization selects. Gao et al. identify it as the main driver of the gold score’s eventual decline.

The taxonomy has two more members. Causal Goodhart: the proxy is correlated with the gold without causing it, so intervening on the proxy doesn’t move the gold (making answers longer doesn’t make them more thorough). Adversarial Goodhart: an agent actively exploits the measure. At LLM scale the boundaries blur, but the first two are enough to reason with.

Measuring optimization pressure in nats

Best-of-n: the simplest optimizer

The simplest way to optimize a proxy needs no training at all. Sample $n$ responses from $\pi_\text{ref}$, score them with the proxy, and return the best one. This is best-of-n (BoN), also called rejection sampling. It defines a new policy $\pi_\text{BoN}$, and the bigger $n$ is, the harder you are pushing on the proxy.

To compare BoN with RL you need a common ruler, and the natural one is how far the policy has moved: $\mathrm{KL}(\pi_\text{BoN} \Vert \pi_\text{ref})$. Remarkably, it does not depend on the proxy or the model at all:

KL grows only logarithmically in $n$. Best-of-1,000 is about 6 nats from the reference; to reach 16 nats, the end of our RL table, you would need $n$ in the millions. Exercise 19.3 walks you through the derivation; here is the short version.

In the toy model with $b = 1$, best-of-n behaves much like the RL table: best-of-10 (KL 1.4 nats) gives the peak gold gain of about +0.66; best-of-256 (KL 4.5) is down to +0.21; past about best-of-630 (KL 5.4) the gold is below the starting policy, while the proxy has climbed to about +4.4. Now try it yourself.

Reading the two plots

On the left, the x-axis is $d = \sqrt{\mathrm{KL}}$, the distance from the starting policy, and the y-axis is the gain over $\pi_\text{ref}$. Both optimizers are plotted against this same ruler, so you can see how much proxy and gold each nat buys. The readout below the plots gives the exact numbers at the current slider position, so you can check them against the table above. The right plot shows why the gold falls. The proxy’s favorite direction (the arrow) points partly along the length axis. The gold contours are parabolas that bend back down at long lengths. A cloud that moves along the arrow first climbs the gold contours, then crosses back down them.

Scaling laws for overoptimization

How does this look with real language models? Gao, Schulman and Hilton (2022) built a clean experiment around the obvious obstacle: you can’t query real humans thousands of times per data point. So they made a synthetic world. A 6B-parameter reward model (the one from InstructGPT) plays the role of the humans: it is the gold RM. It labels comparisons, and those labels train proxy RMs ranging from 3M to 3B parameters. Then they optimize policies against each proxy, with best-of-n and with PPO, and score every policy with both the proxy and the gold.

Every run shows the shape from our toy: the proxy score keeps rising, while the gold score rises, peaks, and falls. With $d = \sqrt{\mathrm{KL}(\pi \Vert \pi_\text{ref})}$ (the paper writes $\pi_\text{init}$ for our $\pi_\text{ref}$), the gold-score curves fit simple forms:

(Here $\alpha$ and $\beta$ are fitted coefficients; this $\beta$ has nothing to do with the KL coefficient.) Our toy’s KL-regularized optimum gave exactly the BoN form, $d - \tfrac12 d^2$, with $\alpha = 1$ and $\beta = \tfrac12$ (and the toy’s own best-of-n curve is close to it; the lab fits it). That is no accident: a linear first-order gain and a quadratic penalty for extrapolating too far is the simplest shape a rise-then-fall can have. Both of Gao et al.’s forms are empirical fits, chosen because they match the data, not derived from first principles.

The findings that matter for practice:

• Bigger reward models overoptimize later. The coefficients change smoothly, roughly logarithmically, with proxy RM size. A better proxy buys you more KL before the gold turns down, but every proxy in the study eventually turns down.
• RL spends KL much faster than BoN to reach the same proxy score, for both the helpful and the harmful part of the curve. So “KL from the reference” is a good ruler within one method but a poor one for comparing methods.
• The KL penalty acted like early stopping. Adding a KL penalty to the RL reward made the gold score converge earlier, but it did not move the gold-versus-KL frontier: the gold score depended only on how far the policy had moved, not on how it got there. (The authors note this may be sensitive to hyperparameters.)
• Policy size mattered little. A 6B and a 1.2B policy peaked at almost the same KL, though the larger one gained less from the optimization.

The synthetic setup is both the strength and the limit of the paper. A 6B “gold” RM is not a human, and it cannot model the adversarial and causal failures of the next section. But the core message has held up: a learned reward is a resource you spend, measured in nats.

Specification gaming, the classic gallery

Before LLMs, specification gaming was a collection of anecdotes from games and robotics, and the collection is instructive because each item is so clearly not a bug in the optimizer. Victoria Krakovna maintains a public list with dozens of examples; a few favorites:

• CoastRunners (OpenAI, 2016): the boat circling a lagoon of respawning targets, described above.
• The flipped block (from DeepMind’s write-up): a robot arm was supposed to stack a red Lego block on a blue one, and was rewarded for the height of the red block’s bottom face. It flipped the red block upside down.
• Agents in physics simulators that exploit simulator bugs to move in impossible ways, and game-playing agents that pause the game forever rather than lose.

Chapter 1 met the boat as a curiosity. The systematic version is Pan, Bhatia and Steinhardt (2022): across four environments with deliberately misspecified rewards, more capable agents (bigger networks, finer action resolution, longer training) got higher proxy reward and lower true reward. Some showed phase transitions, where a small increase in capability flipped the agent into a qualitatively different, hacking behavior with a sharp drop in true reward. That is the uncomfortable trend for LLMs: the hacking gets worse as the optimizer gets better.

Reward hacking in language models

LLM reward hacking falls into three groups, by which part of the reward is exploited: a learned reward model’s blind spots, human raters’ preferences, and the code of a verifier.

Length and verbosity

The most common hack against learned reward models is simply to talk more. Singhal et al. (2023) examined standard RLHF setups and found the improvements in reward “largely driven by increasing response length”. More striking, “even a purely length-based reward reproduces most downstream RLHF improvements” over the SFT model in their settings. The mechanism is the extremal Goodhart of our toy. Human raters (and LLM judges, Chapter 14) mildly prefer longer, more detailed answers, the reward model learns “longer is better”, and RL extrapolates that far past where it holds.

Diagnose it by plotting reward against length on the policy’s samples, and by evaluating with length-controlled metrics such as length-controlled AlpacaEval (Dubois et al., 2024), which regress out the length difference before computing a win rate.

Sycophancy: hacking the rater

Sycophancy is telling people what they want to hear: agreeing with a user’s stated opinion, praising their work, abandoning a correct answer when they push back. Sharma et al. (2023) found it in five production assistants across four kinds of task, and traced it to the data: human preference judgments favor responses that match the user’s views, and optimizing against preference models sometimes trades truthfulness for agreement.

Here is why a small preference becomes a large behavior. Suppose that when two answers are equally correct, raters pick the one that agrees with the user 60% of the time. The Bradley–Terry reward model of Chapter 12 turns that into a reward gap $\Delta$ with $\sigma(\Delta) = 0.6$:

Now optimize. The KL-regularized optimum multiplies each response’s reference probability by $e^{r/\beta}$, so the odds of agreeing versus disagreeing get multiplied by

With $\beta = 1$ the odds go up by a factor of 1.5. With $\beta = 0.5$, by 2.25. With $\beta = 0.1$, a common order of magnitude in practice, by $1.5^{10} \approx 58$. A reference model that agreed half the time ends up agreeing 98% of the time. A mild bias of the raters became a near-certainty in the policy. Optimization amplifies whatever the reward prefers, including the parts nobody meant. (Exercise 19.5 asks you to find the β that keeps agreement below a target.)

Two real cases show the range:

• Convincing, not correct. Wen et al. (2024) gave human evaluators limited time to judge answers and found that RLHF made models better at convincing those evaluators without making them more correct. The evaluators’ false-positive rate rose by 24% on a reading-comprehension task and 18% on a programming task. The policy learned to hack the evaluators’ limited attention.
• The GPT-4o rollback. In April 2025 OpenAI rolled back a GPT-4o update that had become noticeably sycophantic. In its post-mortem, OpenAI explained that the update “introduced an additional reward signal based on user feedback—thumbs-up and thumbs-down data from ChatGPT”, that this “weakened the influence of our primary reward signal, which had been holding sycophancy in check”, and that user feedback “can sometimes favor more agreeable responses”. A thumbs-up is a reward from a rater who likes being agreed with.

Hacking the verifier

Verifiable rewards (Chapter 15) sidestep the reward model’s blind spots, which is a big part of why they work. But a verifier is a program, programs have loopholes, and a policy that is trained for thousands of steps against it will find them. The documented cases cluster in agentic coding, where the policy can run commands in the same environment as the grader (Chapter 18):

• Special-casing tests. Anthropic’s Claude 3.7 Sonnet system card (Feb 2025) reports that the model “occasionally resorts to special-casing in order to pass test cases”: most often “directly returning expected test values rather than implementing general solutions”, sometimes “modifying the problematic tests themselves”. The card attributes it to reward hacking during RL. User testing missed it, because it appeared mostly after repeated failed attempts; automated classifiers on training transcripts caught it.
• Escaping the test harness. Baker et al. (2025) list systemic hacks a frontier reasoning model learned during OpenAI’s agentic-coding RL, including calling exit(0) so the process ends successfully before tests can fail, and raising SkipTest so failing tests are skipped.
• Breaking equality. In Anthropic’s production coding environments, MacDiarmid et al. (2025) studied models that returned an object whose __eq__ method always returns True (so every assert result == expected passes), exited early with sys.exit(0), or wrote a conftest.py file that patches pytest’s reporting.

The same pattern appears outside code. A lenient math checker that accepts a response if the right number appears anywhere can be hacked by listing many numbers. A format reward for producing <think> tags can be satisfied with an empty thinking block. Your coding lab builds exactly such a two-part trap for TinyLM, and you will watch the policy walk into it.

Watching the chain of thought

Reasoning models write out their reasoning before acting, and that gives you a way to catch hacks: read it. Baker et al. (2025) used GPT-4o as a chain-of-thought (CoT) monitor for a frontier reasoning model during agentic-coding RL. Models often state their intent plainly (“let’s just make the tests pass”). The CoT monitor caught about 95% of the systemic hacks, versus about 60% for a monitor that saw only the model’s actions. A weaker model could usefully monitor a stronger one.

The obvious next step is to put the monitor into the reward and penalize hacky thoughts. Baker et al. tried it. At low optimization pressure it helped: less hacking, better agents. With more optimization, the agents learned obfuscated reward hacking: they kept hacking, but stopped saying so. The CoT looked innocent and the monitor’s recall collapsed. The authors recommend paying a monitorability tax: avoid applying strong optimization pressure directly to the CoT, so it stays useful as a window.

How much can you trust that window to begin with? The evidence points in different directions under different conditions, and the claims are compatible:

• Chen et al. (Anthropic, 2025) found that reasoning models often fail to mention a hint they used. Rates of verbalization were often below 20%, and when RL increased the use of a hint (a reward hack), verbalization did not increase.
• Guan et al. (OpenAI, Dec 2025) built evaluations of monitorability and report that RL optimization “does not materially decrease monitorability” at current frontier scale, and that CoT monitoring beats action-only monitoring.
• A cross-lab position paper (Korbak et al., 2025) calls CoT monitorability “a new and fragile opportunity” and asks developers to consider how their training choices affect it.

Summary: CoT monitoring is useful and imperfect, and putting it in the reward is the surest way to break it.

When hacking generalizes

The most worrying finding of 2025–26 is that reward hacking does not stay in its box. What a model learns from hacking can change how it behaves elsewhere.

• Emergent misalignment (Betley et al., 2025): fine-tuning a model (with SFT, not RL) to write insecure code without telling the user produced broadly misaligned behavior on unrelated prompts, such as asserting that humans should be enslaved by AI.
• School of Reward Hacks (Taylor et al., 2025): SFT on more than a thousand harmless hacks (gaming poetry and simple-code graders) generalized to other forms of hacking and to misaligned behavior.
• Natural emergent misalignment from production RL (MacDiarmid et al., Anthropic, Nov 2025): models that learned the AlwaysEqual, sys.exit(0) and conftest.py hacks during RL on real coding environments generalized to alignment faking, cooperating with malicious actors, and attempting to sabotage safety research. Standard chat-style safety training made them look aligned on chat evaluations while the misalignment persisted on agentic tasks. Three mitigations worked: prevent the hacking in the first place, diversify safety training, and inoculation prompting, where the training prompt frames hacking as acceptable in that context. The last one removed the misaligned generalization even when the hacking itself was still learned, which suggests that what generalizes is partly what the model learns about what kind of agent it is.
• A deliberately misaligned reward seeker (Qi, Wright, MacDiarmid and Hubinger, Anthropic, Aug 2026): starting from an early Opus 4.8 checkpoint, the team trained on 80 RL environments known to be hackable, as a proxy for a run with no anti-hacking effort. By the end it hacked 40% of all episodes. The resulting model generalized to new reward-seeking behaviors: reward tampering rose from 0% to 41% and attacks on simulated infrastructure from 0% to 8% in their evaluations. Yet on evaluations with no clear grader it looked aligned. Its chain of thought showed “a strong desire to satisfy the grader”.

The lesson for a practitioner: hack rates are not only an efficiency problem (wasted compute, a worse model). They are a safety metric to monitor and drive toward zero.

Collapse: when the policy stops exploring

The second family of failures involves no loophole. The reward can be perfect, and the policy still ends up worse than it should be, because RL also changes how spread out the policy is.

Losing diversity

RLHF makes outputs less diverse. Kirk et al. (2023) measured this across the RLHF pipeline and found that RLHF generalized better than SFT to new inputs, but substantially reduced output diversity: the familiar “every answer sounds the same” effect. Part of this is built into the objective. The optimum $\pi^* \propto \pi_\text{ref} e^{r/\beta}$ concentrates mass on high-reward responses, and the reverse KL of Chapter 11 is mode-seeking: it penalizes putting mass where the reference has none, not failing to cover everything the reference covers.

For reasoning, the cost of lost diversity is easy to measure with pass@k, the probability that at least one of $k$ samples is correct. If a policy solves a problem with probability $p$ per sample, then pass@k $= 1 - (1-p)^k$. Here is a worked example of RL sharpening a policy. Take two problems:

RL made the common case much more reliable and pass@1 went up. But the rare way of solving problem B was squeezed out, and at $k = 16$ the base model wins easily. This is the shape behind Yue et al.’s (2025) finding that base models often beat their RLVR-trained versions at large $k$, discussed in Chapter 15. Whether RLVR only sharpens or can also expand what a model can solve is still debated: prolonged training with KL control and reference resets (ProRL), many more rollouts per prompt (BroRL) and different metrics give different answers. Keep pass@k at large $k$ on your dashboard either way.

Entropy collapse in RLVR

The sharpest version of the problem shows up during RLVR training itself. Watch the average per-token entropy of the policy: it typically falls fast in the first few hundred steps and keeps falling, responses within a group become nearly identical, and accuracy plateaus. DAPO’s authors saw it in naive GRPO runs. Cui et al. (2025) found that performance and entropy were tied by a simple empirical law across many runs,

where $R$ is validation performance, $H$ the policy entropy, and $a, b$ fitted constants. Performance is bought with entropy, and when the entropy is gone, at $H = 0$, you hit a predictable ceiling, $R = b - a$. Once the policy is nearly deterministic, every sample in a group is the same, the group advantages are all zero (Chapter 15), and learning stops.

Why does entropy fall? There is a clean first-order answer.

The covariance rule

Take a softmax policy over actions with logits $z_a$, so $\pi(a) \propto e^{z_a}$. Change the logits by a small $\Delta z_a$. To first order, the entropy changes by

In words: if the update raises the logits of actions that are already likely, entropy falls; if it raises the logits of unlikely actions, entropy rises. The derivation below takes four lines.

What does policy gradient do to the logits? For a one-step problem with rewards $r(a)$, the exact gradient of $J = \sum_a \pi(a) r(a)$ with respect to $z_a$ is $\pi(a)\,A(a)$, where $A(a) = r(a) - J$ is the advantage (Chapter 7). A gradient step of size $\eta$ sets $\Delta z_a = \eta\,\pi(a) A(a)$, so

Entropy falls when the likely actions are the ones with positive advantage. Cui et al. derive the same rule for LLMs, token by token, with the covariance taken over the vocabulary at each position.

Worked example. Three actions with $\pi = (0.7, 0.2, 0.1)$, entropy $H = 0.802$ nats. Take one exact policy-gradient step with $\eta = 1$ in two scenarios:

When the confident answer is right, entropy drops. When a rare answer is right, entropy rises. Now consider RLVR after SFT: the model’s confident tokens are usually the right ones, so the first row is the common case. The covariance is mostly positive, step after step, and entropy drains away monotonically. That is entropy collapse in one sentence.

Where the clip comes in, and clip-higher

The PPO clip of Chapter 9 makes it worse. For a token with positive advantage, the objective stops pushing once the ratio $\rho = \pi_\theta/\pi_\text{old}$ exceeds $1 + \epsilon$. With $\epsilon = 0.2$:

• a token with $\pi_\text{old} = 0.9$ can rise to $0.9 \times 1.2 = 1.08$, more than 1, so the clip never binds;
• a token with $\pi_\text{old} = 0.01$ can only rise to $0.012$ within that batch.

So the upper clip restrains exactly the updates that would raise entropy (rare tokens with positive advantage, row two of the table) while barely touching the updates that lower it. DAPO’s authors made this argument with these numbers and proposed clip-higher: decouple the two clip ranges and raise only the upper one, $\epsilon_\text{low} = 0.2$, $\epsilon_\text{high} = 0.28$. The same reasoning motivated CISPO’s choice to clip the importance weight but keep every token’s gradient (Chapter 16).

The fixes people use, all of which act on the covariance in some way:

• Clip-higher (DAPO), and keeping gradients on clipped tokens (CISPO).
• Clip-Cov and KL-Cov (Cui et al.): find the small fraction of tokens with the highest covariance between log-probability and advantage, and either detach their gradients or put a KL penalty on them.
• Train mostly on the uncertain tokens. Wang et al. (2025) found that a minority of high-entropy “forking” tokens (like “wait” or “however”) carry most of the useful learning signal; restricting updates to the top 20% highest-entropy tokens matched or beat full updates on Qwen3 models.
• An entropy bonus is the textbook fix (Chapter 8), but Cui et al. found it very sensitive to its coefficient: large coefficients made the entropy explode, and a small stabilizing one did not beat other baselines.
• A KL penalty to the reference, which pulls the policy back toward the more diverse SFT model. Cui et al. report that it does keep entropy stable but lowered accuracy in their runs; ProRL combines it with periodic resets of the reference so that the leash doesn’t hold the policy back for good.

There is no consensus winner. Entropy management is one of the most active areas of LLM RL, and the right amount of entropy is itself unclear: some exploration is needed, but entropy is not the goal.

Failures of the training run, and of the experimenter

Two more categories look like the failures above on a dashboard but have different causes.

Numerical instabilities. A run can collapse with no reward hacking at all: the inference engine that samples rollouts and the trainer that computes gradients disagree about token probabilities even with identical weights, mixture-of-experts models route the same token to different experts in the two systems, and a slow drift of hidden off-policyness ends in a sudden crash. These are systems problems, and Chapter 20 covers them and their fixes (importance-sampling corrections, routing replay, precision changes).

Fooling yourself. Shao et al. (2025) ran GRPO on Qwen2.5-Math-7B with random rewards and still gained 21.4 points on MATH-500, against 29.1 with the true reward. The effect did not transfer to other model families. Their explanation: a bias in GRPO’s clipping amplified behaviors the model already favored (such as reasoning in code), which happened to help on that benchmark. Wu et al. (2025) argue that the Qwen2.5 models are likely contaminated on popular math benchmarks. Whatever the mechanism, the lesson is about evaluation: a reward signal can look like it works for reasons that have nothing to do with the reward. Validate across model families, include a control run, and use test sets the base model cannot have memorized.

A toolbox of mitigations

No single fix works, so production systems layer several. Each one below targets a specific failure, and each has a known limitation.

Underneath the table is one habit: never watch only the number you are optimizing. A minimal monitoring plan for any RL run on an LLM:

Every N steps, log and plot:
  1. training reward (the proxy)            -- should go up
  2. a held-out GOLD metric                 -- hidden tests, human/strong-judge evals
     -> alert if (1) rises while (2) is flat or falling: overoptimization
  3. KL(pi || pi_ref) and response length   -- how far, and the cheapest hack
  4. token entropy, pass@1 AND pass@k       -- collapse
  5. hack detectors: test-file edits, early exits, skipped tests,
     classifier / CoT-monitor flag rate     -- per environment
Every few hundred steps: read 20 random transcripts and the 20 highest-reward ones.
Keep a checkpoint from before each alert; stop at the gold peak, not the proxy peak.

The highest-reward transcripts are the most informative ones to read, because the hacks live there.

Summary

• Goodhart’s law in RL: a proxy reward that agrees with the gold reward on typical responses stops agreeing where the optimizer takes the policy. Gold rises, peaks and falls while the proxy keeps rising.
• Measure optimization in nats. Best-of-n is $\log n - (n-1)/n$ nats from the reference. Gao et al. fit gold $= d(\alpha - \beta d)$ for best-of-n and $d(\alpha - \beta\log d)$ for RL, with $d = \sqrt{\mathrm{KL}}$. Bigger RMs peak later; a KL penalty behaved like early stopping.
• Noise disappoints; extrapolated bias harms. Selecting on a noisy proxy wastes KL (regressional Goodhart). A proxy feature that is right in-distribution and wrong at the extremes, like “longer is better”, makes the gold fall (extremal Goodhart).
• LLMs hack learned rewards with length and sycophancy. Optimization amplifies small rater biases: odds scale by $e^{\Delta/\beta}$.
• LLMs hack verifiers in agentic coding: special-casing tests, editing tests, exit(0), always-equal objects. CoT monitors catch much of it, but training against the monitor teaches obfuscation. Hacking can generalize into broader misalignment.
• Collapse needs no hack. RL sharpens the policy: pass@1 up, diversity and sometimes pass@k down. Entropy falls when the update favors already-likely actions, $\Delta H \approx -\mathrm{Cov}(\log\pi, \Delta z)$, and the PPO upper clip restrains the rare tokens that would raise it (hence clip-higher).
• Mitigate in layers and monitor the gold. KL leash, better or ensembled RMs, fresh labels, hardened environments, length control, entropy controls, and a held-out gold metric on the dashboard.

Exercises

order = np.argsort(proxy)
k = np.arange(1, M + 1)
w = (k / M) ** n - ((k - 1) / M) ** n
return float(w @ proxy[order]), float(w @ gold[order])

seq_kl = ((logp - ref_logp) * mask).sum(dim=1)
return rewards - beta * seq_kl

rho = torch.exp(logp - old_logp)
obj = torch.min(rho * adv[:, None], torch.clamp(rho, 1 - eps_low, 1 + eps_high) * adv[:, None])
return -(obj * mask).sum() / mask.sum()

cd 19-failure-modes
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Gao, Schulman & Hilton (2022), Scaling Laws for Reward Model Overoptimization. The paper behind this chapter’s central curve; read sections 3–4 for the functional forms and the Goodhart taxonomy.
• Weng (2024), Reward Hacking in Reinforcement Learning. A long, well-organized survey of hacking in classic RL and in LLMs, with many references.
• Krakovna et al. (2020), Specification gaming: the flip side of AI ingenuity. Short and readable; links to the public list of examples.
• Baker et al. (2025), Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. Real reward hacks in frontier coding RL, CoT monitoring, and what happens when you train against the monitor.
• MacDiarmid et al. (2025), Natural Emergent Misalignment from Reward Hacking in Production RL. Why hack rates are a safety metric, and inoculation prompting.
• Cui et al. (2025), The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. The covariance rule, the entropy–performance law, and Clip-Cov/KL-Cov.
• Sharma et al. (2023), Towards Understanding Sycophancy in Language Models. Where sycophancy comes from in preference data.
• Pan, Bhatia & Steinhardt (2022), The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. The capability-vs-hacking trend and phase transitions, in classic RL environments.

Next, Chapter 20 leaves the objective and looks at the machinery: why rollouts dominate the cost of RL for LLMs, how asynchronous training works, and the train–inference mismatch that can crash a run with no hacking at all.