RL with verifiable rewards: reasoning models and GRPO

Here is a puzzle. You have a small language model that answers 37+48= correctly about half the time. You have no teacher who can show it better solutions, and no reward model trained on human opinions. You have one thing: a three-line program that checks whether an answer is right. Is that enough to make the model better at arithmetic?

Yes, and that is the subject of this chapter. In Chapter 12 the reward was a neural network trained to imitate human preferences. Such a network can be fooled, so RLHF needs a leash, a critic, and a lot of care. When the reward is a program that checks the answer (exact match for a math problem, unit tests for code), much of that machinery becomes optional. In September 2024 OpenAI’s o1 showed what large-scale RL on checkable problems can do. In January 2025 DeepSeek-R1 published a recipe anyone could follow: a strong base model, a checker, and an algorithm called GRPO that fits in twenty lines. By the end of this chapter you will have written those twenty lines yourself and watched them push your tiny model from 51% to over 80% accuracy.

Why checkable answers changed things

From “think step by step” to learned reasoning

Language models got better at multi-step problems long before RL entered the picture. Chain-of-thought (CoT) prompting (Wei et al., 2022) showed that if the prompt contains worked examples that reason step by step before answering, large models do the same and get far more math problems right. Kojima et al. (2022) found that appending “Let’s think step by step” to the question helps even without examples. The reasoning was there; prompting just drew it out.

The next step was to train on reasoning. A simple and influential version is STaR (Zelikman et al., 2022): sample many step-by-step solutions, keep the ones whose final answer is correct, fine-tune on them, repeat. If you did Exercise 7.8 in Chapter 7, you recognize this: fine-tuning on your own successes is REINFORCE with 0/1 rewards and no baseline. It already is reinforcement learning with a checker, in its plainest form.

Then, in September 2024, OpenAI announced o1. OpenAI says o1 was trained with large-scale reinforcement learning to think in a long chain of thought before answering, and reports that its performance kept improving both with more RL training compute and with more thinking time at inference. That second axis, test-time compute, was new: a model that gets more accurate the longer it is allowed to think. OpenAI did not publish the algorithm, the rewards or the data. The open community spent the next four months guessing.

What “verifiable” means

A verifiable reward is a reward computed by a program that checks the response against a known answer or specification, rather than by a learned model of someone’s judgment. Examples:

• Math with a final answer: extract the answer (say, the contents of \boxed{...}), normalize it, compare with the reference. Reward 1 if equal, else 0. The lab in Chapter 14 builds a robust version of this.
• Code: run the program against unit tests. Reward 1 if all pass (or the fraction that pass).
• Formal proofs: a proof checker such as Lean accepts or rejects the proof.
• Format: a regular expression checks that the response has the required structure.

Training with such rewards is called reinforcement learning with verifiable rewards (RLVR), a name coined by the Tülu 3 report (Lambert et al., 2024). For our TinyLM the verifier is a single line: 1.0 if response == answer else 0.0.

Why does this matter so much? Recall the central worry of RLHF (Chapter 12, and Chapter 19 in full): a learned reward model is an imperfect proxy, trustworthy only near the data it was trained on. Push a policy hard against it and the policy finds the proxy’s blind spots (verbose answers, flattery, confident nonsense) and the true quality falls. That is why RLHF keeps a KL leash to the reference model and stops early.

A verifier has no blind spots of that kind. There is no direction in response space where “the checker thinks it’s right but it isn’t” grows without limit. So you can optimize much harder, for much longer, which is exactly what reasoning models do: thousands of RL steps instead of a few hundred.

The other consequence: the KL leash matters less. DeepSeekMath’s GRPO used a KL coefficient $\beta = 0.04$. DeepSeek-R1-Zero used $\beta = 0.001$ and periodically reset the reference model to the current policy. Many 2025 recipes dropped the KL term entirely, and as of October 2026 the default in Hugging Face TRL’s GRPOTrainer is $\beta = 0$. Chapter 16 follows that thread.

GRPO: compare each answer with its siblings

A worked example first

Take one prompt, $x$ = 37+48=, and sample $G = 4$ responses from the current model. (The group size $G$ is the number of responses sampled per prompt.) Check each one:

Which responses should the model make more likely, and by how much? GRPO’s answer: compare each response with the others in its group. Compute the group’s mean and standard deviation,

and give each response the group-relative advantage

Here that is $+1, -1, -1, +1$. The two correct answers are pushed up, the two wrong ones pushed down, with equal strength.

Now a harder prompt, where only one of four responses is right: rewards $1, 0, 0, 0$. The mean is $0.25$ and the std is $\sqrt{0.25 \times 0.75} = 0.433$, so

The rare success gets a big push up. Each of the three common failures gets a small push down. The advantages still sum to zero.

Finally, suppose all four responses are right: rewards $1, 1, 1, 1$. Every response equals the mean, so every advantage is zero. The same holds if all four are wrong. A group in which every response gets the same reward teaches the model nothing. Hold on to that; it shapes everything about how RLVR is run in practice.

Why “compare with your siblings” is the right idea

You have seen this move before. In Chapter 7 you learned that the policy gradient

is unbiased for any baseline $b$ that does not depend on the sampled response, and that a good baseline (close to the expected reward) slashes the variance. In Chapter 8 the standard choice was the value function, $b = V(x)$ (not exactly the variance-minimizing baseline, as Exercise 7.6 showed, but close and meaningful), which makes $r - V(x)$ the advantage: how much better this response was than what the policy gets on average from this prompt.

For a 0/1 verifiable reward, $V(x) = \mathbb{E}_{y \sim \pi_\theta}[r(x, y)]$ is the probability $p(x)$ that the model solves the prompt. How would you estimate it? PPO-based RLHF trains a critic, a second network (usually as big as the policy) that predicts values token by token (Chapter 12). GRPO does something much simpler: it samples $G$ answers and uses their average reward as a Monte Carlo estimate of $V(x)$. That is the group mean. No critic to train, no critic to store, no critic to get wrong.

Dividing by the std is GRPO’s other ingredient. It makes the advantages dimensionless: whatever the reward scale, each group’s advantages have mean 0 and standard deviation 1. That is convenient, but it is a choice, not a law. It also changes how much each prompt counts. Every group that has any signal gets advantages of the same overall size (their squares always sum to $G$). So a prompt the model solves 1 time in 8 is pushed almost as hard as one it solves 4 times in 8: the average $\lvert\hat A\rvert$ is 0.66 versus 1.0. Without the std division it would be 0.22 versus 0.5. Liu et al. (2025) argue this is a bias toward too-easy and too-hard prompts and propose dropping the std (“Dr. GRPO”). You can see the effect in the demo below and work it out in Exercise 15.4. Chapter 16 weighs the arguments.

Groups that teach nothing

If a prompt is solved with probability $p$, all $G$ responses get the same reward with probability

Read the table from both ends.

• Too-hard prompts teach nothing. If the model never solves a prompt, every group is all zeros, every advantage is zero, and the model gets no gradient toward solving it. RL can only reinforce what the model already sometimes does. This is why R1-style training starts from a strong base model, and why the choice of training problems (difficulty, curriculum) matters so much.
• Too-easy prompts waste compute. As training succeeds, more prompts move toward $p \approx 1$ and more groups come back all-correct. You paid for $G$ generations and got no gradient. DAPO’s dynamic sampling filters such groups out and samples more prompts to fill the batch (Chapter 16).
• Bigger groups help at the extremes. A prompt with $p = 0.05$ yields signal 34% of the time with $G = 8$, but 96% of the time with $G = 64$. DeepSeekMath used $G = 64$; DeepSeek-R1-Zero used $G = 16$.

From advantages to a loss

Every token of response $y^{(i)}$ gets the same advantage $\hat A^{(i)}$. The simplest version of GRPO is then REINFORCE with this advantage, averaged over tokens:

Here $\lvert y^{(i)}\rvert$ is the number of tokens in response $i$, and the log-probability of the whole response is the sum of token log-probs (Chapter 11). Minimizing $\mathcal{L}$ raises the log-probability of every token in the positive-advantage responses and lowers it in the negative ones.

Let’s see what this does in our example. Tokenized, the four responses are 8 5 <eos>, 7 5 <eos>, 8 1 5 <eos>, 8 5 <eos>: lengths 3, 3, 4, 3. Each token’s log-probability gets the gradient weight $\hat A^{(i)} / (G\,\lvert y^{(i)}\rvert)$:

Now look at the first token, chosen right after =. Three responses started with 8 and one with 7. The token 8 at that position collects $+\tfrac{1}{12} + \tfrac{1}{12} - \tfrac{1}{16} \approx +0.10$: pushed up, though the wrong answer 815 also started with it. The token 7 collects $-0.083$. Nobody told the algorithm that the first digit of 815 was fine and the second one was the mistake. That credit assignment emerges only statistically, across many groups in which good and bad answers share some tokens and differ in others. It is crude, which is why process rewards that score individual steps exist (Chapter 17). It also works remarkably well at scale.

Notice too that the long wrong answer is pushed down less per token ($-0.063$ versus $-0.083$) simply because it is longer. That is a side effect of dividing by $\lvert y^{(i)}\rvert$, and Liu et al. (2025) argue that it lets wrong answers drift longer during training. Chapter 16 dissects this “length bias” and the alternatives.

Adding PPO’s clip

Generating responses is the expensive part of RL for LLMs (Chapter 20), so you want to take more than one gradient step per batch. As soon as you do, the samples come from an older policy $\pi_\text{old}$ (written $\pi_{\theta_\text{old}}$ in papers), and you need the importance ratio and the clip from Chapter 9. Per token,

and the token’s surrogate objective is PPO’s

Everything from Chapter 9 carries over: at $\rho = 1$ the gradient equals the REINFORCE gradient above; once a positive-advantage token’s ratio exceeds $1 + \epsilon$ (or a negative-advantage token’s ratio falls below $1 - \epsilon$), its gradient is zero and the update stops pushing it. In our example with $\epsilon = 0.2$, if after one inner step the probability of 5 in response 1 has risen from 0.40 to 0.50, its ratio is 1.25. The advantage is positive and $1.25 > 1.2$, so on the next inner step that token contributes no gradient.

With a single gradient step per batch, $\rho \equiv 1$ and the clip never activates. That is the setting of the DeepSeekMath experiments (one policy update per batch of samples) and of TRL’s default. In that case GRPO’s policy term is exactly REINFORCE with a group baseline.

Adding the KL leash, in the loss

GRPO keeps a penalty for drifting from the reference model $\pi_\text{ref}$ (usually the starting checkpoint), but puts it in the loss rather than in the reward. Classic RLHF subtracts $\beta \log\frac{\pi_\theta}{\pi_\text{ref}}$ from each token’s reward, which mixes the KL into the advantages. DeepSeekMath adds it as a separate term instead, saying this avoids complicating the advantage computation. The per-token estimator is Schulman’s $k_3$ (Chapter 11):

Three things make $k_3$ attractive. It is never negative, since $u - \log u - 1 \ge 0$ for $u > 0$. It is unbiased for $\mathrm{KL}(\pi_\theta \Vert \pi_\text{ref})$ when $y_t$ is sampled from $\pi_\theta$, because $\mathbb{E}_{\pi_\theta}[\pi_\text{ref}/\pi_\theta] = 1$. And it is zero exactly when the two models agree on the token.

A numeric check: if the policy gives a token probability 0.6 and the reference gives 0.5, then $u = 0.5/0.6 = 0.833$ and $k_3 = 0.833 - \log 0.833 - 1 = 0.0157$. Its derivative with respect to $\log\pi_\theta$ is $1 - \pi_\text{ref}/\pi_\theta = 0.167$. Since the loss contains $+\beta k_3$, gradient descent nudges this token’s log-probability down, back toward the reference. Where the policy has fallen below the reference, the push is up. Where they agree, there is no push at all.

The full objective

Put the three pieces together and you have GRPO exactly as the DeepSeekMath paper (Shao et al., 2024) writes it, translated to our notation:

with $\mathbb{D}_\text{KL}$ evaluated per token by $k_3$, and, for an outcome reward, $\hat A^{(i)}_t = \hat A^{(i)}$ for every $t$. In words: sample a group per prompt from the current policy; give each response its group-relative advantage; for every token, take PPO’s clipped surrogate minus a KL penalty; average over each response’s tokens, then over the group; maximize.

The DeepSeek-R1 paper writes the same objective more compactly, with the ratio of whole-sequence probabilities $\pi_\theta(o_i \mid q)/\pi_{\theta_\text{old}}(o_i \mid q)$. Implementations, including the reference code for this chapter’s lab, use the per-token form.

GRPO, one iteration
1. Draw P prompts x_1..x_P from the dataset.
2. For each prompt, sample G responses with the current policy (call it π_old).
3. Score every response with the verifier: r = 1 if correct, else 0.
4. Per group: A_i = (r_i − mean(r)) / (std(r) + tiny). All tokens of y_i share A_i.
5. Compute once, without gradients: log π_old and log π_ref of every sampled token.
6. Repeat μ times (μ = 1 to a few):
     ρ   = exp(log π_θ − log π_old)                        per token
     obj = min(ρ·A, clip(ρ, 1−ε, 1+ε)·A) − β·k3(π_θ, π_ref)  per token
     loss = −mean over responses( mean over that response's real tokens(obj) )
     one gradient step on θ
7. The updated policy becomes π_old for the next iteration.

That’s all of it. No reward model, no value network, no GAE. Compare that with the four models of PPO-RLHF in Chapter 12: policy, reference, reward model and critic. With a verifier and the KL term, GRPO needs two networks, the policy and a frozen reference. With $\beta = 0$ it needs only the policy. Dropping the critic matters for memory: the critic is a second network about as large as the policy, with its own gradients and optimizer states.

Settings in the wild

One more difference hides in the defaults. TRL’s default loss_type is not the per-response averaging of the DeepSeekMath formula above. It averages over all tokens in the batch (DAPO’s choice), and you have to ask for loss_type="grpo" to get the original. Why anyone would change it is a Chapter 16 story.

One historical detail is easy to miss: the paper that introduced GRPO did not use a verifier. DeepSeekMath trained a reward model and used GRPO as a cheaper replacement for PPO, reporting MATH accuracy rising from 46.8% to 51.7% on top of an instruction-tuned 7B model. The combination of GRPO with purely rule-based rewards is what DeepSeek-R1 made famous a year later. The R1-Zero numbers come from the revised R1 paper (arXiv v2 and the Nature version), which gives more hyperparameters than the January 2025 preprint.

DeepSeek-R1-Zero and R1: the recipe goes public

R1-Zero: RL straight from a base model

DeepSeek-R1-Zero skipped supervised fine-tuning entirely. Starting from the pretrained DeepSeek-V3-Base, it ran GRPO with a fixed prompt template:

A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively […]

The reward was the sum of two rules: an accuracy reward (is the final answer right, checked by matching a boxed answer for math or running test cases for code) and a format reward (is the thinking enclosed in the think tags). DeepSeek deliberately used no neural reward model, writing that neural reward models “may suffer from reward hacking in the large-scale reinforcement learning process”.

What happened, as the paper reports it:

• Accuracy climbed a long way. On AIME 2024 (competition math), pass@1 rose from 15.6% to 71.0% over training in the January 2025 preprint (77.9% in the revised version), and to 86.7% with majority voting over many samples.
• Responses got longer on their own. The average response length on training problems grew steadily, from hundreds to thousands of tokens. Nothing in the reward asked for length. The model learned that spending more tokens on checking and re-trying earns more reward: it learned to use test-time compute.
• Reflection appeared. The paper reports behaviors such as revisiting earlier steps and trying alternative approaches, and shows an “aha moment” in which an intermediate checkpoint interrupts its own solution with “Wait, wait. Wait. That’s an aha moment I can flag here.” and starts over.
• Readability suffered. R1-Zero’s reasoning often mixed languages and was hard to read.

R1: four stages

R1-Zero proved the point; DeepSeek-R1 made a usable model. The pipeline has four stages:

1. Cold-start SFT. Fine-tune the base model on “thousands” of long chain-of-thought examples written in a readable format (reasoning, then a summary). This fixes readability and, the paper says, gives a better starting point than the raw base model.
2. Reasoning RL. The same large-scale GRPO as R1-Zero on math, code, science and logic problems, plus a language-consistency reward: the fraction of words in the chain of thought that are in the target language. The paper notes this slightly lowers benchmark scores but makes the output readable.
3. Rejection sampling + SFT. Sample many responses from the RL checkpoint, keep only correct (and readable) ones, about 600k reasoning samples, add about 200k non-reasoning samples (writing, factual QA, translation), and fine-tune the base model again on these 800k examples. This is the STaR idea, with a much stronger generator.
4. RL for all scenarios. A final RL stage mixing rule-based rewards for reasoning with learned reward models for helpfulness and harmlessness on general prompts.

The pattern of alternating RL (to discover good behavior) with SFT on the model’s own filtered outputs (to consolidate it) recurs in many later pipelines (Chapter 21).

Distillation: the cheap way to a small reasoner

DeepSeek also fine-tuned six smaller open models (Qwen2.5 and Llama, from 1.5B to 70B parameters) on the 800k samples generated by R1. This is pure SFT, no RL, and the resulting “R1-Distill” models reason far better than their bases. The paper then asks the obvious question: could RL alone have done the same for a small model? They ran large-scale RL on Qwen-32B-Base for over 10,000 steps. It reached roughly the level of QwQ-32B-Preview (47.0% versus 50.0% pass@1 on AIME 2024), while the distilled 32B model scored 72.6%. Their conclusion: distilling a stronger model into a smaller one is economical and effective, while pushing the frontier still needs stronger base models and larger-scale RL.

Kimi k1.5, the same week

R1 was not alone. Moonshot AI’s Kimi k1.5 report appeared the same week, also describing long-chain-of-thought RL without Monte Carlo tree search, value functions or process reward models. Its policy update starts from the KL-regularized optimum $\pi^* \propto \pi_\text{old}\, e^{r/\tau}$ that you met in Chapter 10 and Chapter 11, and turns it into a loss whose gradient is again a policy gradient with the mean sampled reward as the baseline, plus a squared penalty on the log-ratio to the previous policy. Different derivation, same core move: sample several answers to a prompt and compare each with their average. It also added a length penalty to keep responses from growing without bound.

Does RL teach new reasoning? The pass@k debate

pass@k

pass@k is the probability that at least one of $k$ independent samples solves the problem, averaged over problems. pass@1 is ordinary accuracy. pass@k for large $k$ measures coverage: can the model solve this problem at all, if you let it try many times? To estimate it without bias, draw $n \ge k$ samples per problem, count the $c$ correct ones, and use

the probability that a random subset of $k$ of your $n$ samples contains at least one correct answer (Chen et al., 2021).

The question

GRPO only reinforces answers the model already produces sometimes. So does RL teach the model to solve problems it could not solve before, or does it just make it produce, more reliably, solutions it could already find? DeepSeekMath noticed the issue first: in their experiments RL improved majority-vote accuracy but not pass@k at large $k$, and they read this as RL making the output distribution more robust rather than raising fundamental capability.

Yue et al. (2025) tested this systematically, across model families, RL algorithms and math, code and visual-reasoning benchmarks. Their findings: RLVR-trained models win at small $k$, but the base models reach higher pass@k at large $k$; the problems RL-trained models solve are largely ones the base model could already solve with enough samples; several popular RL algorithms performed similarly; and distillation from a stronger model, unlike RL, can introduce genuinely new reasoning patterns.

A toy example shows how this can happen. Take two problems. The base model solves problem A with probability 0.6 using one strategy, and problem B with probability 0.05 using a rarer strategy. RL sharpens the model toward the common strategy: problem A goes to 0.98, and, because the policy’s entropy collapses, the rare strategy that cracked problem B fades and problem B drops to 0.01.

(For problem A, $1 - 0.4^{64}$ and $1 - 0.02^{64}$ are both 1 to many decimal places, which is where the leading 1 comes from.)

RL helped at $k = 1$ and hurt at $k = 64$: the curves cross. Your lab reproduces this in miniature. In our run of the lab’s passk.py (300 held-out problems, 64 samples each), GRPO raised the TinyLM’s pass@1 from 0.53 to 0.83, while its pass@64 ended slightly below the SFT model’s (0.977 versus 0.983), and it never solved 7 problems where SFT never solved 5. The crossover fell between $k = 8$ and $k = 16$.

The other side

The debate is not settled, and the counter-evidence is serious:

• Longer, more careful RL can expand coverage. ProRL (Liu et al., NVIDIA, 2025) trained for thousands of steps with KL control, periodic reference resets and a diverse task mix, and reports gains at all $k$, including on tasks where the base model failed for every $k$ they tried.
• Answer-only pass@k can mislead. A base model that guesses a right answer through wrong reasoning counts as a success. Wen et al. (2025) propose scoring pass@k only when the reasoning is also correct, and report that by this measure RLVR does extend the reasoning boundary.
• Benchmarks and base models can mislead. Shao et al. (2025) found that GRPO with random or even incorrect rewards substantially improved Qwen2.5-Math-7B on MATH-500, but gave little or no gain on other model families. Wu et al. (2025) argue that Qwen2.5 models are likely contaminated on common math benchmarks, and that on a clean synthetic task only accurate rewards help. Claims about what RL “teaches” should be checked on more than one model family and on uncontaminated data.

A fair summary as of 2026: at the scales most papers study, RLVR mostly sharpens (it raises the probability of good solutions the model can already find), and it can narrow coverage as entropy falls. Whether and when it expands what the model can do depends on training length, exploration, data and how you measure. That is an open question, and Chapter 21 returns to it. For practice, the lesson is concrete: watch pass@k at large $k$ and the policy’s entropy, not just pass@1.

Watching it train

The group view tells you what happens to one prompt. Training happens on thousands. The demo below runs GRPO on a pool of 24 prompts whose difficulty increases from left to right. Each prompt $j$ is solved with probability $p_j = \sigma(w + u_j - d_j)$, where $d_j$ is a fixed difficulty, $u_j$ a parameter specific to that prompt, and $w$ a shared skill parameter that every prompt’s gradient moves, a stand-in for the way a neural network’s weights are shared across problems. Every iteration samples $G$ answers per prompt, computes group advantages, and takes one policy-gradient step on the logits.

This toy has the same qualitative shape as your lab’s training curves and as reports from large runs: accuracy rises, the share of wasted all-correct groups rises with it, and progress on the hardest problems waits until the model occasionally solves them. It also shows why the dataset is a hyperparameter. Problems too hard for the current model are dead weight until the model improves. Problems too easy are dead weight forever.

Why GRPO works, and where it strains

Why it works:

• The baseline is free. The group mean is a Monte Carlo estimate of $V(x)$ built from samples you generate anyway. No critic, no value-learning instability, half the trained parameters of PPO-RLHF.
• It suits sparse, binary, end-of-sequence rewards. A per-token critic must learn to predict a 0/1 outcome from every prefix, a hard regression problem. Comparing complete answers to the same prompt sidesteps it.
• It inherits PPO’s safety rails (the clip) for batch reuse, and a KL leash when you want one.

Where it strains (each is a later chapter):

• Wasted groups at both ends of difficulty, and sensitivity to the data mix (dynamic sampling, curricula: Chapter 16).
• Normalization choices (std division, length averaging) that quietly reweight prompts and responses (Chapter 16).
• Crude credit assignment: every token gets the same advantage (Chapter 17).
• Entropy collapse: the policy becomes more deterministic, which helps pass@1 and can hurt exploration and pass@k (Chapter 19). Your lab’s entropy curve falls steadily.
• Long rollouts dominate the cost, which pushes real systems toward asynchronous, slightly off-policy training (Chapter 20).
• Multi-turn tasks with tools and environments need more than one response per prompt (Chapter 18).

Summary

• Verifiable rewards are computed by a program that checks the answer (exact match, unit tests, proof checkers). They lack the fuzzy blind spots of learned reward models, so RL can optimize much harder. They are not unhackable: verifier bugs get exploited.
• GRPO samples $G$ responses per prompt and gives each the advantage $\hat A^{(i)} = (r^{(i)} - \text{mean}(r))/\text{std}(r)$, shared by all its tokens. For rewards $1, 0, 0, 1$ that is $+1, -1, -1, +1$.
• The group mean is a Monte Carlo estimate of the value baseline $V(x)$ from Chapter 7–8, so GRPO needs no critic. The loss adds PPO’s per-token clipped ratio and a $k_3$ KL penalty to $\pi_\text{ref}$ in the loss.
• Groups where every answer gets the same reward give zero gradient. This happens with probability $p^G + (1-p)^G$, so too-hard and too-easy prompts are wasted, and RL can only reinforce answers the model sometimes produces.
• DeepSeek-R1-Zero showed that GRPO with only accuracy and format rewards, applied directly to a strong base model, raises reasoning accuracy a lot and grows long chains of thought. R1 added cold-start SFT, a language-consistency reward, rejection-sampling SFT and a final general RL stage. Distilling R1 into small models beat running RL on them directly.
• The pass@k debate: RLVR clearly improves pass@1, but at large $k$ base models often match or beat RL-trained ones, suggesting RL mostly sharpens existing abilities. Whether longer or better-designed RL expands them is still open.

Exercises

# Task 2: reshape to (P, G); population std; eps keeps all-equal groups at 0
r = rewards.view(-1, G)
adv = r - r.mean(dim=1, keepdim=True)
if scale_std:
    adv = adv / (r.std(dim=1, unbiased=False, keepdim=True) + eps)
return adv.reshape(-1)

# Task 3
d = ref_logp - logp
return torch.exp(d) - d - 1.0

# Task 4
ratio = torch.exp(logp - old_logp)
A = adv[:, None]                                   # same advantage for every token
surrogate = torch.minimum(ratio * A, torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * A)
per_token = surrogate - beta * k3_kl(logp, ref_logp)
per_response = (per_token * mask).sum(1) / mask.sum(1).clamp(min=1)
loss = -per_response.mean()

cd 15-rlvr-grpo
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Shao et al. (2024), DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Where GRPO comes from. Section 4 has the objective, the outcome- and process-supervision variants, and a “unified paradigm” that writes SFT, rejection-sampling fine-tuning, DPO, PPO and GRPO as one gradient with different coefficients.
• DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, published in revised form in Nature 645, 633–638. R1-Zero, the four-stage R1 pipeline, distillation, and an honest “unsuccessful attempts” section on PRMs and MCTS. The revised version adds many hyperparameters.
• Yue et al. (2025), Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. The careful pass@k study behind the “sharpening” view. Read it alongside ProRL (Liu et al., 2025) for the other side.
• Liu et al. (2025), Understanding R1-Zero-Like Training: A Critical Perspective. Questions the aha moment, finds length and difficulty biases in GRPO, and proposes Dr. GRPO. The best bridge to Chapter 16.
• Kimi Team (2025), Kimi k1.5: Scaling Reinforcement Learning with LLMs. The parallel recipe from the same week: mirror-descent derivation, length penalty, partial rollouts for long contexts.
• Lambert et al. (2024), Tülu 3: Pushing Frontiers in Open Language Model Post-Training. Names RLVR and places it in a fully open SFT → DPO → RLVR pipeline, with data and code.
• Schulman (2020), Approximating KL Divergence. The short blog post that introduced the $k_1$, $k_2$, $k_3$ estimators GRPO uses.
• Hugging Face TRL, GRPO Trainer documentation. When you want to run GRPO on a real model: reward-function conventions, loss variants, and logged metrics such as frac_reward_zero_std. The lab’s colab_grpo_trl.md is a starting point.

Next, Chapter 16 takes GRPO apart knob by knob (baseline, normalization, aggregation, clipping, KL) and shows that RLOO, Dr. GRPO, DAPO, GSPO and CISPO are all the same estimator with different settings.