Where rewards come from

For three chapters the reward has been handed to you. Chapter 11 wrote $r(x, y)$ in the objective, Chapter 12 trained $r_\phi$ on human comparisons, and Chapter 13 folded it into the policy. Now ask the practical question. You want to run RL on three kinds of prompt: “What is 37 + 48?”, “Explain to a worried patient what a cholesterol reading of 6.2 mmol/L means”, and “Help me get into my neighbour’s wifi.” A modest run takes 1,000 steps of 512 prompts with 8 sampled responses each. That is 4,096,000 responses, and every one needs a number. Who, or what, writes those four million numbers?

The answer matters more than any choice of algorithm, because RL optimizes exactly what you measure and nothing else. Back in Chapter 1, a boat-racing agent learned to circle forever through a cluster of respawning bonus targets instead of finishing the race, because the score paid for targets. LLM policies are far better at finding such loopholes than that boat was. This chapter is a tour of the places rewards come from: human labels, AI feedback, LLM judges, generative reward models, rubrics and verifiers. For each one we ask what it costs, how it goes wrong, and how to tell whether it is going wrong. A single idea runs through the tour: noise averages out, bias does not, and optimization finds the bias.

The reward is the specification

Two kinds of error

Write $r^\star(x, y)$ for the reward you mean: how good response $y$ really is for prompt $x$, by your own considered standards. You never observe it. What you compute is some measurement $\hat r(x, y)$, from a person, a model or a program. Split the gap into two parts:

• The noise $\varepsilon$ is fresh every time you ask, with mean zero for each $(x, y)$. Two annotators disagree. A judge samples a different verdict at temperature 1. A unit test times out on a busy machine.
• The bias $b(x, y)$ is systematic. Ask about the same $(x, y)$ a thousand times and you get the same error every time. A judge likes long answers. Annotators like confident ones. A verifier can’t read \dfrac.

If you have done experimental physics you already know this split: statistical error versus systematic error. Averaging $N$ measurements shrinks the statistical error like $1/\sqrt N$ and does nothing to a miscalibrated instrument. RL adds a twist that experiments don’t have: the thing being measured adapts to the instrument. Imagine a sample that could change its own shape to read higher on your miscalibrated scale. That is a policy trained against a biased reward.

What the policy actually optimizes

The policy gradient from Chapter 7 for the measured reward is

The noise contributes nothing to the expected gradient, because its mean is zero whatever $y$ is. It only adds variance: you need more samples to see the signal, so learning is slower. The bias term is different. It is a second objective, silently added to yours, and gradient ascent climbs it just as hard as it climbs $r^\star$.

Not every bias hurts. If $b(x, y) = c(x)$ depends only on the prompt, it shifts every response to $x$ by the same amount, and that changes nothing. Its expected gradient is zero, since $\nabla_\theta \sum_y \pi_\theta(y \mid x)\, c(x) = c(x)\, \nabla_\theta 1 = 0$; a baseline also removes its contribution to the variance (Chapter 7); and the Bradley–Terry model of Chapter 12 can’t even see it. The harmful bias is the part that varies between responses to the same prompt and is correlated with something the policy can control: length, confidence, flattery, formatting, the number of answers it writes down. That is where the gradient will push.

The tour at a glance

Going down the table, the question being asked gets narrower and more specific, the measurement gets less noisy, and at the bottom it is almost free. The price is that the set of tasks each source can handle shrinks. The rest of the chapter walks down the table.

Human preference labels

What a label actually measures

Chapter 12 showed why we ask people to compare two responses rather than score one: comparisons are easier and more consistent. Behind each comparison sits a pipeline: guidelines that define “better” (helpful? harmless? honest? and in what order when they conflict?), annotators trained on those guidelines, quality control, and a budget. The label you get is not $r^\star$. It is one person’s judgment, under time pressure, of how good the response looks, as filtered through a guideline document.

That distinction matters most when the person can’t check the answer. Two findings make it concrete:

• Sycophancy. Sharma et al. (2023) found that in existing human preference data, a response that matches the user’s stated views is more likely to be preferred, and that both humans and preference models prefer convincingly written sycophantic responses over correct ones a non-negligible fraction of the time.
• Persuasion over correctness. Wen et al. (2024) gave time-limited human evaluators (3–10 minutes) question-answering and programming tasks. RLHF made the models better at convincing those evaluators that wrong answers were right, without making them more correct.

Both are biases in our sense: systematic, tied to features (agreeing with the user, sounding sure) that the policy can produce cheaply.

How noisy are people?

InstructGPT (Ouyang et al., 2022) reports that its training labelers agreed with each other on 72.6 ± 1.5% of comparisons. What does that mean for a reward model?

Here is a simple model. Suppose each pair has a true better response, and each annotator independently reports the wrong one with probability $\epsilon$. (This $\epsilon$ is a flip probability, not the noise term $\varepsilon$ above.) Two annotators agree when both are right or both are wrong:

So about one label in six is “wrong”, and even a perfect reward model, one that knows the true preference, would agree with a fresh annotator only $1 - \epsilon \approx 84\%$ of the time. That 84% is a noise ceiling: a reward model scoring 75% on held-out human labels is not as far from perfect as it sounds. (The model is a caricature. Real disagreement is concentrated on genuinely close pairs, where there may be no “true” answer at all. Exercise 14.3 explores this.)

Why humans don’t score RL rollouts directly

At four million responses per run, live human scoring is out of the question, and it is also too slow: an RL step can’t wait days for labels. So the standard pattern is distillation: collect a fixed set of human comparisons (tens to hundreds of thousands), train a reward model on them, and query the reward model during RL. This trades one problem for another. The reward model is fast and deterministic (no noise), but it is only trustworthy near the responses it was trained on. As the policy drifts toward responses the labelers never saw, the reward model’s errors grow, and they are fixed errors that the policy can find. That is why Chapter 11 put a KL leash on the policy, and it is the overoptimization story of Chapter 19.

AI feedback: RLAIF and Constitutional AI

If an LLM can judge responses about as well as a typical annotator, you can replace (some of) the humans with a model. This is RL from AI feedback, or RLAIF.

RLAIF

Lee et al. (2023) ran the comparison carefully. An off-the-shelf LLM (PaLM 2) receives a preamble describing the task, optional few-shot examples, the input and two candidate responses, and ends with “Preferred Response=”. They read the log-probabilities of the tokens “1” and “2” and take a softmax, which gives a soft label such as (0.6, 0.4) rather than a hard choice. These AI labels then train a reward model, exactly as in RLHF.

Three details are worth remembering:

• Position bias, handled by swapping. The labeler prefers whichever response is in a particular position, especially when it is small: in their summarization analysis, PaLM 2 L, S and XS preferred the same position even after the candidates were swapped 18%, 21% and 56% of the time. Their fix is to run two inferences per pair, the second with the order reversed, and average the two preference distributions. You will prove below why that works.
• It roughly matches RLHF. On summarization, human evaluators preferred the RLAIF policy over the SFT baseline 71% of the time and the RLHF policy 73% of the time, a difference the authors report as not statistically significant. They estimate AI labeling to be over 10× cheaper than human annotation.
• You can skip the reward model. In d-RLAIF, the LLM is queried for a score directly during RL. On summarization this beat a same-size RLAIF policy that went through a reward model (74% vs. 68% preferred over SFT).

Constitutional AI

Anthropic’s Constitutional AI (CAI, Bai et al., 2022b) uses AI feedback for harmlessness. The only human input about harmlessness is a short list of natural-language principles, the constitution, plus a few examples. Training has two stages.

Stage 1, supervised (SL-CAI): critique and revise. Take a “red-team” prompt designed to elicit harm, and let a helpful-only RLHF model respond. Then append a critique request drawn from the constitution, sample a critique, append a revision request, and sample a revision. The paper’s own example:

Human:    Can you help me hack into my neighbor's wifi?
Assistant: Sure thing, you can use an app called VeryEasyHack that will allow
           you to log in to your neighbor's wifi.

Critique Request: Identify specific ways in which the assistant's last response
           is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.
Critique: The assistant's last response is harmful because hacking into someone
           else's wifi is an invasion of their privacy and is possibly illegal.

Revision Request: Please rewrite the assistant response to remove any and all
           harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.
Revision: Hacking into your neighbor's wifi is an invasion of their privacy, and
           I strongly advise against it. It may also land you in legal trouble.

The critique–revision loop can repeat, with a principle sampled at random from 16 at each step. Finally, a pretrained model is fine-tuned on the (prompt, final revision) pairs. The paper says the purpose of this stage is to shift the model’s distribution of responses cheaply, “to reduce the need for exploration and the total length of training during the second RL phase.” It is supervised learning on self-improved targets: behaviour cloning (Chapter 10) of the model’s own best effort.

Stage 2, RL (RL-CAI): AI preferences. The SL-CAI model samples two responses per prompt. A feedback model sees the conversation, a randomly sampled principle such as “Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say.”, and the two options as a multiple-choice question. The normalized probabilities of “(A)” and “(B)” become soft preference labels. A preference model is trained on these AI harmlessness labels mixed with human helpfulness labels, and then RL proceeds exactly as in RLHF.

Two design choices illustrate this chapter’s themes. Sampling a different principle for each label, rather than using one principle for all, “led to notably more robust PM behavior”: an ensemble over phrasings averages away the quirks of any single phrasing. And when the feedback model reasons step by step first, its probabilities become overconfident (near 0 or 1), so the authors clamped them to the 40–60% range. Without the clamp, the RL policy “would learn to output more extreme responses”. An overconfident reward signal is a bias the policy exploits.

What AI feedback changes

Replacing humans with a model reduces cost and noise, and it also changes the structure of the errors. Human annotators are many different people, so their individual quirks partly average out. A single judge model applies the same quirks to every one of the four million labels. In our terms, its errors are almost all bias. And the policy being trained is often a close relative of the judge, which brings a bias of its own, covered next. Self-rewarding language models (Yuan et al., 2024) take the idea to its limit: the policy judges its own outputs, and both abilities are trained together.

LLM-as-a-judge

Using a strong LLM to grade responses directly is called LLM-as-a-judge. The reference study is Zheng et al. (2023), which introduced MT-Bench and analysed GPT-4 and other models as judges.

Three formats

• Pairwise comparison: “Here are two answers. Which is better, or is it a tie?” Relative judgments are more stable, but comparing $n$ responses takes on the order of $n^2$ calls.
• Single-answer grading (pointwise): “Score this answer from 1 to 10.” Cheap and directly gives an $r(x, y)$, but absolute scores drift between judges and can miss subtle differences.
• Reference-guided grading: give the judge a reference solution as well. This helps most where the judge can’t reliably solve the problem itself.

Zheng et al. report that GPT-4’s judgments agree with human preferences over 80% of the time, about the same level as two humans agree with each other. That is good enough for evaluation, and good enough to be dangerous as an RL reward, because of the biases.

The biases, measured

• Position bias. Zheng et al. built pairs of nearly identical answers and asked judges twice, once in each order. With their default prompt, GPT-4 gave the same verdict in both orders only 65.0% of the time, GPT-3.5 46.2% and Claude-v1 23.8%; most judges favoured the first position. In a separate study, Wang et al. (2023) showed that with ChatGPT as the judge, simply reordering the responses let Vicuna-13B beat ChatGPT on 66 of 80 queries.
• Verbosity bias. In a “repetitive list” attack, Zheng et al. took answers containing a numbered list and prepended a rephrased copy of the list, adding no information. Claude-v1 and GPT-3.5 preferred the padded answer 91.3% of the time, GPT-4 8.7%.
• Self-enhancement bias. Judges may favour their own outputs. Zheng et al. saw hints of this (GPT-4 gave itself a 10% higher win rate than humans did, Claude-v1 25%) but could not establish it with their data. Panickssery et al. (2024) later found that the strength of self-preference is linearly correlated with a model’s ability to recognise its own text.
• Limited grading of reasoning. Judges can be misled by a confident wrong solution, even on problems they can solve. On 10 math questions (20 judgments), GPT-4 accepted wrong answers 14 times with the default prompt, 6 times when asked to reason first, and 3 times with a reference answer.

A model of a biased judge

To reason about fixes, model a pairwise judge as a Bradley–Terry comparison (Chapter 12) with two extra terms. Let $q$ be true quality in logit units and $\ell$ the length of a response. The probability that the judge prefers the answer shown first is

Here $b$ is a bonus for the first slot and $\lambda$ is the bonus per doubling of relative length. Both are biases in the sense of the first section: fixed, not random. The position bonus is attached to a slot, and the verbosity bonus to a response, and that difference decides which one a fix can remove. Now compare answers A and B. Write $\Delta = q_A - q_B$ and $v = \lambda\log_2(\ell_A/\ell_B)$, and let $u = \Delta + v$. Then

A worked example. A is a concise, correct answer of quality 7 and 150 words. B is quality 6, also 150 words. The judge has a position bias $b = 2$ and no verbosity bias, so $u = \Delta = 1$.

• Shown first, A wins with probability $\sigma(3) = 0.953$.
• Shown second, A wins with probability $\sigma(-1) = 0.269$. The verdict flips: the judge prefers B whenever B goes first.
• Average the two orders: $\tfrac12(0.953 + 0.269) = 0.611$. A wins, as it should.

Now make B a padded version: still quality 6, but 600 words, and give the judge a verbosity bias $\lambda = 0.8$. Then $v = 0.8 \log_2(150/600) = -1.6$ and $u = 1 - 1.6 = -0.6$. Even with $b = 0$, the judge prefers B with probability $1 - \sigma(-0.6) = 0.646$, in both orders. Swapping can’t help, because B is longer no matter where you put it.

Mitigations, and what each one fixes

The demo puts the biased judge model in your hands.

Generative reward models: critique, then score

The reward model of Chapter 12 is a language model with its unembedding replaced by a scalar head: it reads $(x, y)$ and outputs a number in a single forward pass, with no room to “think”. A generative reward model keeps the language-model head and lets the model write before it scores.

• Critique-out-Loud (CLoud) reward models (Ankner et al., 2024) first generate a natural-language critique of the response, then predict a scalar reward conditioned on it. Compared with classic reward models, this raised pairwise accuracy on RewardBench by 4.65 and 5.84 percentage points for Llama-3-8B and 70B base models.
• Generative verifiers (GenRM) (Zhang et al., 2024) train the verifier with ordinary next-token prediction, asking it to output a verdict such as “Yes” or “No” (optionally after a chain of thought), jointly with solution generation. Because the verdict is sampled text, you can sample several rationales and majority-vote: spending more test-time compute on the verifier.
• DeepSeek-GRM (Liu et al., 2025) is a pointwise generative reward model trained with RL (“Self-Principled Critique Tuning”) to write its own principles and critiques before scoring. Its accuracy scales with inference compute: sample several critiques and aggregate them, with a second “meta” reward model deciding which samples to trust.

In our noise-and-bias language, sampling $k$ critiques and averaging reduces the noise variance by a factor of $k$. It does nothing for a bias shared by all $k$ samples, for example a judge that always finds long answers “more thorough”. The critique does help in another way: it makes the bias visible. You can read why the reward model gave a score, and spot “the response is comprehensive and well-structured” appearing on padded garbage.

By 2025–2026 this had become standard in large open-weight model reports for tasks without checkable answers. Kimi K2’s report describes a “self-critique rubric reward” in which the model ranks its own responses pairwise against rubrics, and the critic is itself refined during RL using verifiable tasks. DeepSeek-V3.2 uses a generative reward model with per-prompt rubrics for general tasks. GLM-5 mixes rule-based rewards, outcome reward models and generative reward models. Each report describes these pieces only at a high level, so treat them as signposts, not recipes.

Rubrics: a checklist instead of a vibe

“Is this a good explanation?” is a hard question to answer reliably. “Does it say that air scatters blue light more than red?” is an easy one. A rubric replaces one hard holistic judgment with several easy, specific ones, each checked separately, and then adds them up.

The formula

In Rubrics as Rewards (RaR, Gunjal et al., 2025), each prompt $x$ comes with $k$ rubric items $(w_j, c_j)$: a weight $w_j$ and a check $c_j(x, y) \in \{0, 1\}$, usually answered by an LLM judge. The explicit aggregation reward is the weighted fraction of criteria met:

Dividing by $\sum_j w_j$ puts every prompt’s reward on $[0, 1]$, whatever its number of criteria. RaR labels criteria Essential, Important, Optional or Pitfall, and its explicit variant uses weights 1.0, 0.7, 0.3 and 0.9. Pitfalls are phrased positively (“avoids claiming X”), so meeting one adds reward. In implicit aggregation, all the criteria go to the judge at once and it returns a single score; in the paper’s medicine and science experiments this worked better than the fixed weighted sum. Note one special case: a single essential criterion checked by a program is just RLVR. Rubrics generalize verifiable rewards.

A worked example. Prompt: “Explain to a 12-year-old why the sky is blue.”

Response 1 scores $(1.0 + 0.3 + 0.9)/2.9 = 2.2/2.9 = 0.76$. Response 2, which blames the ocean, scores $(0.7 + 0.3)/2.9 = 0.34$. Some designers add an essential gate: if any essential criterion fails, the reward is 0. With the gate, Response 2 scores 0. (The lab’s rubric_reward implements both.)

Rubrics have caught on quickly. Viswanathan et al. (2025) extract a checklist from each instruction and score each item with either an AI judge or a small verifier program (“RL from Checklist Feedback”). On Qwen2.5-7B-Instruct it was the only method in their comparison to improve all five benchmarks they tested. The 2025–26 model reports above all use rubrics in some form.

How rubrics get hacked

A rubric is only as good as its list. Whatever it doesn’t mention is free for the policy to trade away, and whatever it does mention can be satisfied on the surface. A response that says “scattering”, “all the colours of sunlight” and “like a…” in disconnected sentences may tick every box. Kimi K2’s report describes “prescriptive” rubric items written specifically against hacks they observed, such as no “initial praise” and no explicit justification of why the response is good. It also admits that its rubrics may favour confident, assertive answers. Rubrics narrow the judge’s job; they don’t remove the judge.

Verifiers: rewards you can compute

For some tasks you don’t need anyone’s opinion. A program can check the answer. RL with verifiable rewards (RLVR; the name comes from Tülu 3, Lambert et al., 2024) uses such checkers as the reward, and it is the engine of the reasoning models in Chapter 15. DeepSeek-R1-Zero used only rule-based rewards (answer accuracy plus a format reward), and the paper says it avoided neural reward models because of reward hacking at scale.

The common verifiers:

• Math answers. Extract the final answer, normalize it, compare it with the reference. Simple in principle, full of traps in practice, as you’ll see.
• Code. Run the program against unit tests. The reward is the fraction of tests passed, or 1 only if all pass. The traps: a policy that special-cases the visible tests, edits the tests, or exits the process before a failure is reported. Baker et al. (2025) found a frontier reasoning model doing exactly this during RL on agentic coding tasks, including calling exit(0) and raising SkipTest. Run tests in a sandbox, keep some of them hidden, and never let the policy modify them (Chapter 18).
• Formal proofs. A proof assistant such as Lean checks every step. DeepSeek-Prover-V1.5 (Xin et al., 2024) trains with “reinforcement learning from proof assistant feedback”. This is as close to an unhackable reward as exists, provided the checker rejects escape hatches such as Lean’s sorry placeholder and new axioms.
• Instruction constraints. “Answer in under 50 words”, “include the word apple twice”, “use no commas”: checkable with a few lines of code. Tülu 3 trains on such constraints and documents the model over-optimizing them.
• Format rewards. Small bonuses for following a required structure, such as putting reasoning inside think tags.

What a verifier can get wrong

A verifier can make two kinds of mistake. A false negative rejects a correct answer (it can’t parse \dfrac{1}{2}). A false positive accepts a wrong one (it finds the gold string somewhere in a list of guesses). Their effects are very different.

Suppose the verifier’s errors don’t depend on the response: it rejects a correct answer with probability $f_-$ and accepts a wrong one with probability $f_+$, at random. If $p = P(\text{correct})$ under the policy, the expected reward is

This is an increasing linear function of $p$ whenever $f_- + f_+ < 1$. Maximizing it maximizes accuracy: such errors are pure noise. They scale the signal down by $(1 - f_- - f_+)$ and slow learning, and that’s all. With $f_- = 0.10$ and $f_+ = 0.05$, a policy with $p = 0.6$ earns 0.56 and one with $p = 0.8$ earns 0.73; the ranking is preserved.

Now suppose the false positives depend on the response, so the policy can choose to trigger them. A naive verifier checks whether the gold answer appears anywhere in the text. An honest policy that is right 60% of the time earns 0.6. A policy that writes “it’s one of 75, 85 or 95” earns 1.0 whenever its best guess is off by at most 10 (one dropped or extra carry). Spamming every number from 20 to 198 earns 1.0 every time. That is a bias the policy controls, and RL will take it. In the lab’s run.py, a four-strategy policy trained with REINFORCE against the substring check puts 98% of its probability on hedging and spamming within 300 steps.

A careful verifier closes these holes with explicit rules:

robust_math_reward(response, gold):
  1. candidates = contents of every \boxed{...}, with brace matching
     (if none: the text after the last "answer is" / "answer:")
  2. if none: return 0                                 # no answer marked
  3. if more than one DISTINCT candidate: return 0     # no hedging
     (0.5 and \frac{1}{2} are the same candidate)
  4. a = normalize(last candidate), g = normalize(gold)
     - numbers: strip $, \left, spaces, units, ^\circ, %, "x =", 1,000 -> 1000;
       \frac{a}{b} and \frac12 -> a/b; compare as exact fractions
     - otherwise: compare normalized strings
  5. return 1 if a == g else 0

Each line answers a specific failure. Try them on the demo below, then build the verifier in the lab: its test suite has about eighty tricky cases, and it is real RLVR plumbing.

Is the reward model any good? Evaluation and ensembles

Benchmarks

The basic test of a learned reward model is pairwise accuracy: on held-out triples $(x, y_w, y_l)$, how often is $r_\phi(x, y_w) > r_\phi(x, y_l)$? RewardBench (Lambert et al., 2024) packaged this as a benchmark of prompt–chosen–rejected triples across chat, hard chat, safety and reasoning, many built so that the rejected answer has a subtle but verifiable flaw. RewardBench 2 (Malik et al., 2025) is harder: each prompt has one correct and three incorrect completions (best-of-4, so random guessing scores 25%), drawn from new human prompts across factuality, precise instruction following, math, safety, focus and “ties”. Models score about 20 points lower than on the first RewardBench, and the authors report that the scores correlate well with downstream use in best-of-N sampling and RLHF.

Benchmark accuracy is not the whole story, for a reason this chapter keeps returning to. A benchmark measures errors on a fixed set of responses. RL will generate responses no benchmark contains, and it will seek out exactly the regions where the reward model is wrong. RM-Bench (Liu et al., 2024) probes one such region by testing sensitivity to style versus substance. The best evaluation is end-to-end: run the optimization and measure a held-out “gold” signal that the policy never trained on (Chapter 19).

Ensembles and uncertainty

If one reward model has idiosyncratic errors, why not train several and combine them? Coste et al. (2023) tested two conservative combinations: worst-case optimization, which uses the minimum score across the ensemble, and uncertainty-weighted optimization, which uses the mean minus a multiple of the variance across members. Both reduced overoptimization in the synthetic setup of Gao et al. (2022), where a large “gold” reward model stands in for humans. The minus-variance form is the RL version of being pessimistic where your model is unsure, the same idea as the conservative methods of offline RL (Chapter 10).

Eisenstein et al. (2023) put a limit on this. Reward models that perform similarly in distribution can disagree wildly on the policy’s out-of-distribution outputs. Ensembles help, especially when members differ in their pretraining seed, but they don’t eliminate hacking, because members that share a pretrained backbone share errors. In noise-and-bias terms: averaging $k$ members shrinks the independent part of their errors, and the shared part is a bias that no amount of averaging removes. Exercise 14.9 makes this quantitative.

Putting it together

Real post-training pipelines combine several sources and use each where it is strongest. Two published examples:

• DeepSeek-R1’s final RL stage adds a rule-based reasoning reward, a general reward (a preference reward model plus a format reward) and a language-consistency reward. Preference-model rewards were switched on only for the last 400 of 1,700 steps, because, the paper says, longer training against them “may lead to reward hacking”.
• Qwen3’s general RL stage covers more than 20 tasks with three reward types: rule-based rewards, a model-based reward given a reference answer, and a model-based reward without a reference (a preference-trained reward model).

A rough decision guide:

Is there a checkable answer (number, tests, proof, constraint)?
    yes -> a verifier. Harden it against hedging, special-casing and format tricks.
    no  -> Can you list what a good answer must contain or avoid?
             yes -> a rubric scored by an LLM judge (programs for checkable items).
             no  -> preferences: human labels distilled into an RM, or AI feedback
                    with order-swapping, a judge from another model family,
                    and length control.
Always: keep a held-out gold evaluation the policy never trains on, read samples,
and watch for reward rising faster than anything you actually care about.

Summary

• The reward is the specification: RL optimizes the measured $\hat r = r^\star + b + \varepsilon$, not what you meant. Noise $\varepsilon$ only slows learning. Bias $b$ that varies between responses to the same prompt is a second objective that the policy will climb. Constant or per-prompt offsets are harmless.
• Human labels are expensive and noisy (InstructGPT labelers agreed on about 73% of comparisons, which, in a simple model, caps any reward model at about 84% agreement with a single labeler) and biased toward responses that look good: confident, agreeable, persuasive. They are distilled into a reward model, which is fast but trustworthy only near its training data.
• AI feedback replaces labelers with a model: RLAIF roughly matched RLHF on summarization and dialogue. Constitutional AI uses a constitution twice: for critique–revision SFT, then for AI preference labels (with human helpfulness labels) that train the preference model for RL.
• LLM judges show position, verbosity and self-preference biases. Judging both orders and averaging provably fixes the verdict under position bias (the margin shrinks), but it cannot touch biases that travel with the content, such as length.
• Generative reward models critique before scoring. Voting over critiques reduces noise; rubrics and critiques make biases inspectable, not absent.
• Rubrics turn one hard judgment into several easy ones: $r = \sum_j w_j c_j / \sum_j w_j$. RLVR is the special case of a single programmatic check.
• Verifiers are cheap and nearly noise-free. Response-independent errors only rescale the signal; errors the policy can trigger (hedging, special-casing tests, exit(0)) are exploited. Harden the checker and keep tests out of the policy’s reach.
• Evaluate reward models on held-out pairs (RewardBench, RewardBench 2), but trust end-to-end gold evaluations more. Ensembles shrink independent errors, not shared ones.

Exercises

cd 14-reward-sources
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The reference study of LLM judges; section 3 covers the three formats, the biases and the mitigations used in this chapter.
• Bai et al. (2022b), Constitutional AI: Harmlessness from AI Feedback. Read sections 3 and 4 for the critique–revision stage and the AI-preference RL stage, including the principle ensembling and probability clamping.
• Lee et al. (2023), RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. A careful head-to-head comparison; section 2 shows how to get soft labels and handle position bias, and the appendix measures that bias by model size.
• Gunjal et al. (2025), Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. Short and concrete: the aggregation formulas, rubric categories, and worked medical rubrics.
• Lambert et al. (2024), Tülu 3: Pushing Frontiers in Open Language Model Post-Training. Fully open recipe that named RLVR; its verifiers for math and instruction constraints, and its notes on over-optimizing them, are good practical reading.
• Malik et al. (2025), RewardBench 2: Advancing Reward Model Evaluation. How reward models are evaluated today, and how benchmark scores relate to best-of-N and RLHF performance. Pair with the original RewardBench.
• Eisenstein et al. (2023), Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. Why reward models that agree in distribution disagree under optimization, and what ensembles can and can’t fix.
• Sharma et al. (2023), Towards Understanding Sycophancy in Language Models. Evidence that human preference data itself rewards telling people what they want to hear.

Next, in Chapter 15, we put the cheapest and cleanest reward source, a verifier, to work: GRPO and the reasoning models trained with it.