The critic-free algorithm zoo

Open three reinforcement-learning papers from 2025 and you will meet three new acronyms: DAPO, Dr. GRPO, GSPO. A month later there are three more. Each paper prints a long objective full of sums, mins, clips and fractions, and claims it fixes something the previous one got wrong. It feels like learning a new algorithm every week.

It isn’t. Almost every method in this family computes the same thing: REINFORCE from Chapter 7, with a baseline taken from a group of samples, applied to the tokens of a response. The papers differ in a handful of choices: what to subtract from the reward, whether to divide by a standard deviation, how to average over tokens, how to clip the importance ratio, whether to keep a KL term, and which samples to keep. This chapter calls those choices knobs. You will learn the master formula once, see what each knob does on a batch of eight responses small enough to compute by hand, and then read every algorithm in the zoo as a setting of the knobs. After that, the next new acronym will take you five minutes to place.

You should know GRPO from Chapter 15 (group sampling, group-normalized advantages, the clipped token-level objective) and PPO’s clip from Chapter 9. We recap what we need.

One estimator, six knobs

Where we start: REINFORCE on a response

Take one prompt $x$ and treat the whole response $y$ as a single action, as in the contextual-bandit view of Chapter 11. The reward $r(x, y)$ arrives at the end. The policy gradient with a baseline $b$ (Chapter 7) is

The log-probability of a response is the sum of its token log-probabilities, so the same gradient, written token by token, is

Read it as an instruction: every token of the response gets the same push, of size $r - b$. Tokens of better-than-expected responses become more likely, tokens of worse-than-expected ones less likely. This is the true gradient. Everything below is a way of estimating it from a finite batch, more cheaply, more stably, or (sometimes) of deliberately estimating something slightly different.

The master formula

Sample $G$ responses $y^{(1)}, \dots, y^{(G)}$ for each prompt. A batch has $N$ responses in total (prompts × $G$). Every critic-free method computes an update of this form:

In words: each token’s log-probability is pushed by the response’s advantage $\hat A^{(i)}$, times a per-token weight $w^{(i)}_t$ that corrects (or clips) for the batch having been sampled by a slightly older policy, times a normalizing constant $c^{(i)}$ that decides how tokens are averaged. Here are the six knobs:

GRPO as a setting of the knobs

Here is GRPO’s objective as the DeepSeekMath paper writes it (their $o_i$ is our $y^{(i)}$, their $q$ is our $x$; Chapter 15 derives it):

Match it against the knobs. Baseline: the group mean. Scale: the group std. Aggregation: $\frac1G\sum_i \frac{1}{\lvert y^{(i)}\rvert}\sum_t$, an average over tokens inside each response, then over responses. Ratio and clip: PPO’s clip on a per-token ratio $\rho^{(i)}_t = \pi_\theta(y^{(i)}_t\mid\cdot)/\pi_\text{old}(y^{(i)}_t\mid\cdot)$. KL: in the loss, with the $k_3$ estimator, $\beta = 0.04$. Data: every group is used. At the first update on a fresh batch, $\pi_\theta = \pi_\text{old}$, every $\rho = 1$, nothing is clipped, and the gradient of the bracket is exactly the master formula with $w = 1$ and $c^{(i)} = 1/(G\lvert y^{(i)}\rvert)$.

Every other algorithm turns one or more of these knobs. Before the zoo, let’s understand each knob on a concrete batch.

The running example: eight responses

Two prompts, $G = 4$ responses each, a verifier that says right (1) or wrong (0). The lengths differ on purpose.

Prompt 1 is the rewards 1, 0, 0, 1 group from Chapter 15: half right. It has a short wrong answer ($y^{(2)}$, 4 tokens) and a long wrong one ($y^{(3)}$, 20 tokens). Prompt 2 is harder: one right answer out of four. The batch has $N = 8$ responses and 90 tokens. The generation budget is $L_\text{max} = 20$ tokens. The demo later in the chapter uses exactly this batch.

Knob 1: the baseline

What a baseline is for

Chapter 7 proved two facts. Subtracting any baseline $b$ that does not depend on the sampled response leaves the gradient unbiased, because $\mathbb E_{y\sim\pi}[\nabla\log\pi(y\mid x)] = 0$. And a good baseline (close to the expected reward) cuts the variance, often enormously. With 0/1 rewards and no baseline, every wrong answer has $r = 0$ and gets no push at all: the policy learns only from its successes. With a baseline, wrong answers get a negative advantage and are pushed down.

The ideal baseline is $V(x) = \mathbb E_{y\sim\pi}[r(x,y)]$, the expected reward for this prompt. PPO learns it with a critic (Chapter 8). For a language model the critic is a second network as large as the policy, and per-token values for reasoning traces are hard to learn. Critic-free methods estimate $V(x)$ from the samples themselves. They differ in how.

Five baselines, computed

None (plain REINFORCE). $\hat A^{(i)} = r^{(i)}$.

Batch mean. Average all rewards in the batch: $3/8 = 0.375$ here. Cheap, but it uses one number for an easy and a hard prompt alike.

Group mean (GRPO). Average the $G$ rewards of the same prompt: $0.5$ for prompt 1, $0.25$ for prompt 2. This is a Monte Carlo estimate of $V(x)$ for this prompt.

Leave-one-out (RLOO). For response $i$, average the other $G-1$ rewards of its group:

The idea goes back to Kool, van Hoof and Welling (2019), “Buy 4 REINFORCE samples, get a baseline for free!”. Ahmadian et al. (2024) brought it to RLHF as RLOO (REINFORCE Leave-One-Out). Because $b^{(i)}$ never looks at $r^{(i)}$, it does not depend on the sampled response $y^{(i)}$, and the estimator is exactly unbiased.

Greedy reward (ReMax, Li et al. 2023). Decode one extra response greedily (always the most likely token) and use its reward as the baseline. One extra generation per prompt instead of a group.

Here are the advantages on our batch, before any scaling:

Look at prompt 2’s lone success. The group baselines give it the biggest advantage in the batch (0.75 or 1.0): a rare success on a hard prompt is the most surprising thing that happened, so it gets the biggest push. The batch mean misses this: it gives the same 0.625 to every right answer, easy prompt or hard.

Group mean versus leave-one-out: a rescaling

Look at the last two columns: the leave-one-out advantages are the group-mean ones multiplied by $4/3$. That is no coincidence. Write $S = \sum_j r^{(j)}$ for the group’s total. Then

The group-mean advantage is exactly $(G-1)/G$ times the leave-one-out advantage, response by response. Since the leave-one-out estimator is unbiased, the group-mean estimator has expectation $\frac{G-1}{G}\nabla J$: it is biased, but only by a constant factor shared by every prompt with the same $G$. A constant factor is absorbed by the learning rate (and almost entirely ignored by Adam). So GRPO’s baseline “includes the sample itself” in a harmless way. Exercise 16.2 asks you to prove the expectation directly.

Knob 2: the scale (to divide by std, or not)

What GRPO’s std does

GRPO divides each group’s centered rewards by the group’s standard deviation. For prompt 1 (rewards 1, 0, 0, 1) the std is 0.5, so the advantages become ±1. For prompt 2 (0, 0, 0, 1) it is $\sqrt{0.25\times0.75} = 0.433$, giving $-0.577$ for the wrong answers and $+1.732$ for the right one. (We use the population std, dividing by $G$. Some implementations use $G-1$; the difference is a constant factor $\sqrt{G/(G-1)}$.)

The intent is sensible: put every prompt’s advantages on the same scale, as in Chapter 8’s advantage normalization. But a group’s std is not a neutral number. With binary rewards and pass rate $p$, the std is $\sqrt{p(1-p)}$. It is largest at $p = 0.5$ and goes to zero as $p$ approaches 0 or 1. Dividing by it up-weights the prompts the model almost always gets right and the prompts it almost always gets wrong.

How much, exactly?

Ask what each prompt contributes to the expected gradient. Without std, using the true mean $p$ as the baseline,

where $p(x)$ is the probability that a sample for prompt $x$ is right. So mean-centering follows the gradient of $\sum_x p(x)$, the expected accuracy, with every prompt weighted equally. With the std, every term is divided by $\sqrt{p(1-p)}$:

The extra factor $1/\sqrt{p(1-p)}$ is a prompt weight:

A prompt the model solves 99% of the time is pushed five times harder per unit of $\nabla p$ than a prompt it solves half the time. Liu et al. (2025) call this the question-level difficulty bias of GRPO and remove the std in Dr. GRPO (“GRPO Done Right”).

Is the difficulty bias good or bad?

Reasonable people disagree, and the evidence is mixed:

• Against the std. Dr. GRPO removes it. Olmo 3’s recipe and DeepSeek-V3.2 use mean-centered advantages without std. Cursor’s Composer 2 report drops it because “small behavioral differences get massively upweighted” when all rollouts are about equally good. You can see why with non-binary rewards: a group with rewards 0.80, 0.81, 0.79, 0.80 has std 0.007, so the advantages become 0, +1.41, −1.41, 0. Noise of one hundredth becomes a full-sized training signal.
• For some reweighting. DAPO and GSPO keep the group std. REINFORCE++ (Hu et al. 2025) argues that per-prompt (“local”) std normalization is biased and unstable when the std is near zero, and divides by one global batch std instead. That keeps the overall scale but weights every prompt the same.
• Go further the other way. MaxRL (Tajwar et al. 2026) argues we should up-weight hard prompts much more, by aiming at $\sum_x \log p(x)$ (the log-likelihood of success) instead of $\sum_x p(x)$. Its gradient is $\nabla p / p$, a weight of 100 at $p = 0.01$ and about 1 at $p = 0.99$. That is asymmetric: hard prompts count, easy ones don’t.
• It may not matter much at the end. ScaleRL (Khatri et al. 2025) fit compute-performance curves for many recipe variants and found that advantage normalization mainly changes how fast a run improves, not the level it eventually reaches.

The three weightings in one picture: mean-only weights every prompt 1; std weights both extremes; MaxRL weights only the hard extreme. Each one is the exact gradient of a different objective ($\sum p$, $\sum 2\arcsin\sqrt p$, $\sum\log p$). “Bias” just means “not the objective you thought you were optimizing”.

Knob 3: aggregation, and the length bias

Three ways to average over tokens

After computing a per-token term (advantage times ratio, clipped or not), you have a ragged array: response $i$ has $\lvert y^{(i)}\rvert$ terms. You must turn it into one number to differentiate. Three choices are in use:

They look like bookkeeping. They are not, because they decide the weight $c^{(i)}$ each token gets, and that weight can depend on the length of the response the token sits in.

The worked example

Take prompt 1’s two wrong answers under GRPO’s advantages ($\hat A = -1$ for both). The short one has 4 tokens, the long one 20. Each token’s weight $c\cdot\hat A$:

Under sequence-mean, each response gets the same total push no matter how long it is, so each token of the long wrong answer is pushed down five times less than each token of the short one. Under the other two, every token is pushed equally, and the long answer, with five times as many tokens, gets five times the total push.

Which one is the policy gradient? Go back to the first section: the true gradient gives every token of response $i$ the same weight $r^{(i)} - b$, with no $1/\lvert y\rvert$. The constant normalizer gives exactly that, up to one global factor. Token-mean is almost that: its normalizer $1/\sum_i\lvert y^{(i)}\rvert$ is shared by the whole batch, so it only rescales the step, by an amount that varies a little from batch to batch. Sequence-mean is the odd one out.

What sequence-mean actually optimizes

There is a cleaner way to see the bias. Under sequence-mean, response $i$ contributes

That is the REINFORCE estimator for a different reward, $\tilde r(y) = (r(y) - b)/\lvert y\rvert$ (treating the baseline as fixed; with the leave-one-out baseline it is exactly independent of $y^{(i)}$). Now look at what $\tilde r$ prefers, with $b = 0.5$:

• A right answer earns $0.5/\lvert y\rvert$: shorter is better. A 100-token right answer earns 0.005; a 1000-token one earns 0.0005.
• A wrong answer earns $-0.5/\lvert y\rvert$: longer is better. A 100-token wrong answer costs 0.005; a 1000-token one costs only 0.0005.

So GRPO as written pays the model to be brief when it is right and to ramble when it is wrong. Liu et al. (2025) call this the response-level length bias: “longer responses are penalized less” when they are incorrect, so incorrect responses grow. It matters whenever the model can tell, mid-response, that it is going wrong, which language models can (they hedge, loop, re-check). Part of the famous growth in response length during R1-Zero-style training may come from this bias rather than from useful reasoning, which is one of the paper’s points.

Dr. GRPO’s fix is to replace each response’s $\lvert y^{(i)}\rvert$ by one constant, the generation budget $L_\text{max}$. The paper also points out that the same per-response length normalization was already present in open-source PPO implementations before GRPO, so this is not a GRPO-only issue.

What the lab shows

The lab for this chapter includes a tiny “language model” built to expose the bias. A response is an answer token followed by filler tokens; the filler never changes the reward, but the policy may learn to emit more filler after some answers than after others. With everything else equal, sequence-mean aggregation (the GRPO preset) roughly doubles the average length of wrong responses within 300 updates (from about 4 tokens to about 8, with noisy spikes past 10), and shortens right responses from about 3.5 tokens to 2.2. Under Dr. GRPO’s constant normalizer and DAPO’s token-mean, both stay flat. The effect is not subtle once length is something the policy controls.

Knob 4: importance ratios and clipping

Why ratios appear at all

If you generated a batch and took exactly one gradient step on it, you would need no ratio: the samples come from the current policy. In practice you reuse a batch for several minibatch updates, and in large systems the samples are also a few versions stale (asynchronous training) and come from an inference engine whose probabilities differ slightly from the trainer’s (Chapter 20). So from the second update on, $\pi_\theta \neq \pi_\text{old}$, and as in Chapter 9 each token’s term is reweighted by the importance ratio

PPO’s clip then keeps the update from running away. Using $\nabla\rho = \rho\,\nabla\log\pi$, the weight a token gets in the master formula under the PPO clip is

Here $\epsilon_\text{low}$ and $\epsilon_\text{high}$ are the lower and upper edges of the clip band; standard PPO and GRPO use one value for both, $\epsilon_\text{low} = \epsilon_\text{high} = \epsilon = 0.2$. A token that has already moved far enough in the direction its advantage wants stops contributing a gradient (the “four cases” of Chapter 9). The methods in this section disagree about three things: where the band should be, whether a clipped token should lose its gradient entirely, and whether the ratio should be computed per token at all.

Clip-higher: give rare tokens room to grow (DAPO)

The symmetric clip treats a ratio of 1.2 the same for every token. In probability, that is very unequal. A token with $\pi_\text{old} = 0.6$ may rise to 0.72 before its gradient switches off: a gain of 0.12. A rare token with $\pi_\text{old} = 0.01$, say the word “Wait” that starts a self-correction, may only rise to 0.012: a gain of 0.002. Rare tokens are exactly the ones whose probability the policy needs to raise to explore new behaviour, and the upper clip throttles them sixty times harder in absolute terms.

DAPO (Yu et al. 2025) argues that this throttle is one cause of entropy collapse, where the policy becomes nearly deterministic early and stops improving (Chapter 19). Its fix, clip-higher, decouples the two edges: $\epsilon_\text{low} = 0.2$ but $\epsilon_\text{high} = 0.28$. In DAPO’s ablation on Qwen2.5-32B, adding clip-higher raised AIME 2024 accuracy from 36 to 38 points. (AIME 2024 is a set of 30 competition-math problems, a standard reasoning benchmark; avg@32 means accuracy averaged over 32 sampled answers per problem, which reduces the noise of a single sample.) The lower edge stays at 0.2 because pushing bad tokens’ probabilities toward zero too fast is the other way to collapse entropy.

CISPO: clip the weight, keep the gradient (MiniMax-M1)

MiniMax’s M1 report observed that low-probability “fork” tokens such as “However”, “Recheck” and “Wait” get large ratios after the first on-policy update and are then clipped out, losing their gradient for all the remaining off-policy updates on that batch (they used 16). Their fix, CISPO (Clipped IS-weight Policy Optimization), does not clip the objective. It clips the importance weight and stops its gradient:

where $\operatorname{sg}$ means stop-gradient: the weight is treated as a constant. In practice only the upper bound is used. The gradient weight is then $w = \min(\rho, 1+\epsilon_\text{high})$, which is never zero: every token keeps a gradient, but no token’s weight can exceed $1 + \epsilon_\text{high}$. This is truncated importance sampling, an old idea in off-policy RL, applied to REINFORCE. MiniMax reports that in a controlled comparison on Qwen2.5-32B, CISPO reached DAPO’s performance with half the training steps.

Compare the two rules on four tokens (with $\epsilon = 0.2$ for PPO, and only an upper bound of 1.28 for CISPO):

PPO’s zeros are where it believes the token has “done enough”. CISPO never believes that; it only refuses to let one token’s weight explode. Note the second row: CISPO caps a weight that PPO would leave unclipped. The two rules are different trade-offs, not one a superset of the other.

Sequence-level ratios (GSPO)

The reward belongs to a whole response, so the “correct” importance weight for a response is the ratio of sequence probabilities, $\pi_\theta(y\mid x)/\pi_\text{old}(y\mid x) = \prod_t \rho_t$. Its trouble is variance: a product of hundreds of ratios near 1 can still land anywhere from 0.01 to 100. GRPO’s per-token ratios dodge that, but each token is then weighted by its own noisy ratio. The GSPO paper argues that this is not a proper importance correction for a sequence-level reward at all, only noise injected token by token.

GSPO (Group Sequence Policy Optimization; Zheng et al. 2025, the Qwen team) takes the principle “the unit of optimization should match the unit of reward” and uses one ratio per response, the geometric mean of its token ratios:

Every token of a response now shares one weight, and the whole response is clipped or not, together. A worked example: three tokens with ratios 1.3, 0.8, 1.0 and $\hat A > 0$. GRPO’s token clip (ε = 0.2) zeroes the first token’s gradient and keeps the other two with weights 0.8 and 1.0. GSPO computes $s = (1.3\times0.8\times1.0)^{1/3} = 1.04^{1/3} = 1.013$ and gives all three tokens the same weight, unless $s$ is outside its band.

That band is tiny. Because $\log s$ is an average per-token log-ratio, it barely moves even when individual tokens move a lot. The GSPO paper uses a clip range of $3\times10^{-4}$ on the left and $4\times10^{-4}$ on the right, against 0.2 and 0.27 for its GRPO baseline. Even so, it reports that GSPO clips about two orders of magnitude more tokens than GRPO, yet trains more efficiently, which the authors read as evidence that GRPO’s token-level gradients are noisy.

The motivation that made GSPO matter in practice is mixture-of-experts (MoE) models. In an MoE, a small change in the weights can route a token to different experts, so a single token’s ratio can jump wildly between $\pi_\text{old}$ and $\pi_\theta$. The GSPO paper reports that GRPO on its MoE models needed Routing Replay (reuse the old policy’s expert choices when computing ratios) to converge, and GSPO did not. The Qwen team credits GSPO for improvements in Qwen3.

Knob 5: the KL leash

Chapter 11 introduced the KL-regularized objective and three per-token estimators of $\mathrm{KL}(\pi_\theta\,\Vert\,\pi_\text{ref})$: $k_1 = \log\frac{\pi_\theta}{\pi_\text{ref}}$, $k_2 = \frac12\big(\log\frac{\pi_\theta}{\pi_\text{ref}}\big)^2$, and $k_3 = \frac{\pi_\text{ref}}{\pi_\theta} - \log\frac{\pi_\text{ref}}{\pi_\theta} - 1$. The zoo uses three placements:

• In the reward. Subtract $\beta\sum_t k_1$ from each response’s reward before computing advantages. This is classic RLHF (Chapter 12) and RLOO.
• In the loss. Add $\beta\,k_3$ per token to the loss, outside the advantage. This is GRPO, with $\beta = 0.04$ in DeepSeekMath. REINFORCE++ puts $k_2$ in the loss instead.
• Dropped. DAPO, Dr. GRPO, CISPO, Magistral, Olmo 3 and ScaleRL train RLVR with no KL term at all. DAPO’s argument: for long chain-of-thought reasoning “the model distribution can diverge significantly from the initial model, thus this restriction is not necessary.”

Why would anyone drop the leash? Its main job in RLHF is to keep the policy where a learned reward model is still trustworthy (Chapter 19). A verifier that checks a final answer cannot be fooled by moving off-distribution in the same way, and reasoning models are supposed to move far from the base model. Dropping the KL also saves a forward pass through the reference model. On the other side, DeepSeek-V3.2 keeps an importance-corrected $k_3$ and says the right strength is domain-dependent (weak or none for math), and NVIDIA’s ProRL keeps the KL and periodically resets the reference model to sustain long training runs.

There is also a subtlety about which placement gives the gradient you think it does. “A Comedy of Estimators” (Shah et al. 2025) analyzed the on-policy gradients: $k_1$ in the reward gives an unbiased gradient of the reverse KL; GRPO’s $k_3$ in the loss gives a biased gradient that nevertheless trains stably; $k_3$ in the reward led to collapse in their runs. The lab implements all three placements so you can check the formulas; Chapter 11 has the estimator math.

Knob 6: which samples enter the batch

Zero-signal groups

If all $G$ responses to a prompt get the same reward, every baseline that uses the group (mean or leave-one-out) gives every advantage exactly 0. The group costs a full round of generation and contributes no gradient. How often does that happen? With pass rate $p$, all right or all wrong has probability $p^G + (1-p)^G$:

As training succeeds, more prompts become easy, and a growing share of the batch is dead weight. Worse, the dead groups still count in the normalizer ($N$ for sequence-mean, $\sum_i\lvert y^{(i)}\rvert$ for token-mean), so they also shrink everyone else’s step.

Dynamic sampling (DAPO) over-samples prompts and filters out groups whose rewards are all equal, refilling until the batch is full of informative groups. In DAPO’s ablation it was the largest single step, from 42 to 50 on AIME 2024 avg@32. Olmo 3 uses a more efficient variant it calls active sampling, and ScaleRL adds “No-Positive-Resampling”: prompts with a pass rate of at least 0.9 are retired from the dataset for good.

Overlong responses

Responses that hit the generation limit are cut off and usually scored as wrong, even if the reasoning was on track. That injects noise: the model is punished for a good but long attempt. DAPO’s overlong filtering masks truncated responses out of the loss. Its soft overlong punishment adds a length penalty that ramps up in a buffer zone of $L_\text{cache}$ tokens before the hard limit:

With DAPO’s settings, $L_\text{max} = 20{,}480$ and $L_\text{cache} = 4{,}096$, responses up to 16,384 tokens are not penalized at all; an 18,000-token response sits 1,616 tokens into the buffer and gets $-1616/4096 \approx -0.39$. In the ablation, overlong filtering and the soft punishment were worth +6 and +3 points. Note this is a reward-design knob more than an estimator knob; Chapter 19 discusses length penalties as a defence against verbosity.

Build an algorithm

Everything so far fits in one picture. The demo shows the running example’s eight responses as rows of tokens. Each cell is one token; its color is the gradient weight $c^{(i)}w^{(i)}_t\hat A^{(i)}$ that multiplies $\nabla\log\pi$ for that token (blue: pushed up; orange: pushed down; hatched: clipped to exactly zero). The bar at the right of each row is the response’s total push. The numbers are relative: the largest per-token weight in the batch reads 1.00, and so does the largest total. Under the bars, three readouts compare the short and long wrong answers, the signal each prompt gets, and how many tokens are clipped. Tap a response’s label (for example “y₂ ✗”) to flip its reward.

A few things to notice as you play:

• On-policy, half the knobs are invisible. With staleness 0 every ratio is 1, so GRPO and GSPO give identical weights, and every clip setting does nothing. Ratio and clip knobs only matter for the second and later updates on a batch, or under asynchronous or mismatched sampling.
• Baselines change who learns from failure. On REINFORCE, wrong answers have no color at all. On every group baseline they are pushed down, and prompt 2’s single success is the brightest row in the batch.
• The std changes the balance between prompts. Compare the “signal per prompt” line for GRPO and Dr. GRPO: the std inflates the 1-of-4 prompt relative to the 2-of-4 prompt. Flip prompt 2’s $y^{(4)}$ to make it a 0-of-4 group and its signal vanishes. Make it 3 of 4 instead and it gets exactly the same signal as 1 of 4: $\sqrt{p(1-p)}$ is symmetric in $p$ and $1-p$.

The zoo, species by species

The comparison table

Each row follows the cited paper (dates are arXiv v1 months). “Seq-mean”, “token-mean” and “constant” are the three aggregations above. “Group” means computed within the $G$ samples of one prompt.

Species notes

RLOO is the cleanest member: leave-one-out baseline, no std, no clip, the KL folded into the reward. Ahmadian et al. argue that the RLHF setting doesn’t need PPO’s machinery: with a good starting policy and a sequence-level reward, the clip rarely fires, and treating the whole response as one action (a bandit) works as well as modeling partial completions with a critic, at a fraction of the cost.

GRPO is the reference point everyone modifies. Its group mean and std came from DeepSeekMath, which used a learned reward model. Its popularity came with DeepSeek-R1 and verifiable rewards (Chapter 15).

Dr. GRPO is a correction, not an invention: drop the two divisions ($\lvert y\rvert$ and the std) that make GRPO optimize something other than expected reward. Its paper also contains useful skepticism about R1-Zero-style training, including evidence that some base models already show “aha”-style self-reflection before any RL.

DAPO is a bundle of four engineering fixes found while reproducing R1-Zero at 32B scale: clip-higher, dynamic sampling, token-level loss, and overlong shaping. Its ablation table is one of the best places to see how much each knob is worth in one setting:

Remember these are cumulative and measured once, in one setting; the order of addition affects how much each step appears to contribute.

CISPO and GSPO both attack the off-policy knob, in opposite directions. CISPO keeps per-token ratios but never discards a token’s gradient. GSPO discards per-token ratios entirely in favour of one per sequence, and clips whole responses. Both appear in ScaleRL’s comparison as having a higher performance ceiling than DAPO’s loss.

REINFORCE++ and ScaleRL prefer a batch-level std: one number for the whole batch, so no prompt is up-weighted for having low variance.

Critics strike back

“Critic-free” is a trend, not a law. VAPO (Yue et al. 2025, ByteDance Seed) is PPO with a value model, made to work for long chain-of-thought: the value network is pretrained, GAE uses different λ for critic and policy, and λ adapts to response length. It reports 60.4 on AIME 2024 from Qwen2.5-32B base, more than 10 points above DAPO in the same setting. VinePPO (Kazemnejad et al. 2024) replaces the learned critic with Monte Carlo value estimates: because a language environment can be reset to any prefix, you can sample a few continuations from the middle of a response and average their rewards, giving unbiased per-step values (Chapter 17 uses the same trick for process rewards). It reports beating PPO and RLOO-style baselines with up to 3× less wall-clock time to peak. The honest summary: group baselines give up per-token credit assignment in exchange for simplicity and memory. When credit assignment within a long response matters, critics (learned or Monte Carlo) can win back the difference, at a cost.

Does the choice matter?

Less than the number of papers suggests, and more than zero. Yue et al. (2025), the pass@$k$ paper (not the VAPO authors), compared six popular RLVR algorithms and found they performed similarly, all far from the base model’s large-$k$ pass@$k$ (Chapter 15 discusses that debate). ScaleRL found that most knobs change how quickly a run reaches its ceiling rather than the ceiling itself, with a few exceptions (the loss type, FP32 precision for the LM head, batch size, generation length). The lab’s TinyLM experiment shows the same flavour at toy scale: REINFORCE, RLOO, GRPO and Dr. GRPO all improve the addition task by similar amounts within noise. Knob choices matter most when they interact with a specific failure: lengths that the model can inflate, entropy that collapses, MoE ratios that explode, prompts that become too easy.

Beyond the knobs: changing the objective

One 2026 line of work changes not the estimator but the target. MaxRL (Tajwar et al. 2026) observes that with binary rewards the log-probability of success expands as

so ordinary RL ($\nabla p = \nabla\,\mathrm{pass@}1$) keeps only the first term of the maximum-likelihood gradient. Its estimator is a one-line change to REINFORCE: average the score function over the successful samples only, dividing by the number of successes instead of by $G$. That turns the per-prompt weight $1$ (mean-centered) into roughly $1/p$, the third of the three weightings compared in Knob 2. The authors report it Pareto-dominates the baselines they tested; it is recent, so treat it as a promising direction rather than settled practice.

Reading the next acronym

When a new critic-free method appears, ask six questions, in order:

1. Baseline:     what is subtracted from r? (none / group mean / leave-one-out / batch / greedy / learned)
2. Scale:        divided by a std? whose? (group / batch / none)
3. Aggregation:  how are token terms averaged? (1/|y| per response, 1/total tokens, constant)
4. Ratio & clip: per token or per sequence? clipped, capped, masked, or soft-gated? which band?
5. KL:           in the reward, in the loss (which estimator?), or absent?
6. Data:         which groups are dropped or refilled? how are truncated responses treated?

Then ask the question that separates a real improvement from a knob shuffle: which failure does the change target, and does the paper show that failure in its baseline? A method that fixes length bias should show lengths growing without the fix. A method that fixes entropy collapse should show entropy collapsing. If the paper only shows a better final number, on one model family, with one seed, read it skeptically (Chapter 21 has a checklist).

Here is the full step every method in the table implements, with the knobs marked:

for each training step:
  1. sample a batch of prompts; for each prompt sample G responses from π_old (the current policy)
  2. score them: r_i = verifier(x, y_i)
     (knob 6) drop groups with all-equal rewards and refill; mask or penalize truncated responses
  3. (knob 5a) if KL in reward: r_i ← r_i − β Σ_t log(π_old/π_ref)
  4. (knobs 1-2) A_i = (r_i − baseline_i) / scale_i
  5. record log π_old(y_t) for every token (and log π_ref(y_t) if KL in loss)
  6. for each of K minibatch updates:
       ratio: ρ_t = π_θ/π_old per token, or s_i per response           (knob 4)
       term_t = clip rule(ρ, A_i)                                      (knob 4)
       term_t ← term_t − β · KL estimate_t     if KL in loss           (knob 5b)
       loss = − aggregate(term)                                        (knob 3)
       gradient step on loss

Summary

• The critic-free family is one estimator: each token’s log-probability gradient is weighted by an advantage, an off-policy weight, and an aggregation constant. Algorithms differ in six knobs: baseline, scale, aggregation, ratio and clip, KL, and data.
• Baselines: group mean (GRPO) is exactly $(G-1)/G$ times leave-one-out (RLOO), a harmless rescaling; leave-one-out is unbiased; no baseline means wrong answers are never pushed down.
• Dividing by the group std reweights prompts by $1/\sqrt{p(1-p)}$, up-weighting nearly-always-right and nearly-always-wrong prompts (Dr. GRPO’s difficulty bias), and amplifies noise when rewards barely differ. Batch std, no std, and MaxRL’s $1/p$ are alternatives, each the gradient of a different objective.
• Sequence-mean aggregation optimizes $(r-b)/\lvert y\rvert$: it pays the model to be short when right and long when wrong. Token-mean and constant normalizers give every token equal weight and remove the bias.
• Clipping matters only off-policy. Clip-higher (DAPO) lets rare tokens grow; CISPO caps the importance weight but never zeroes a gradient; GSPO uses one geometric-mean ratio per response with a tiny band, which stabilizes MoE training.
• KL can go in the reward, in the loss, or nowhere; most 2025 RLVR recipes dropped it, and the estimator and placement change which gradient you actually get.
• Data knobs (dynamic sampling, overlong filtering) were among the largest single gains in DAPO’s ablation, because zero-variance groups and truncated responses waste compute or inject noise.
• Measured head to head, many variants perform similarly; the knobs pay off when they target a failure you can actually see in your run.

Exercises

cd 16-policy-gradient-zoo
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Liu et al. (2025), Understanding R1-Zero-Like Training: A Critical Perspective. Dr. GRPO. The clearest statement of GRPO’s length and difficulty biases, plus useful skepticism about what R1-Zero-style training really teaches. Start with the section on GRPO’s biases.
• Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Four fixes (clip-higher, dynamic sampling, token-level loss, overlong shaping) and an ablation table that prices each one.
• Ahmadian et al. (2024), Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. RLOO, and the argument that PPO’s machinery is mostly unnecessary for LLM fine-tuning. Short and well argued.
• Zheng et al. (2025), Group Sequence Policy Optimization. GSPO: sequence-level ratios, the gradient comparison with GRPO, and the MoE motivation.
• MiniMax (2025), MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. The CISPO section, with the “fork tokens” diagnosis, plus the FP32 LM-head fix for train–inference mismatch (Chapter 20).
• Khatri et al. (2025), The Art of Scaling Reinforcement Learning Compute for LLMs. ScaleRL. Hundreds of thousands of GPU-hours spent asking which knobs change the asymptote and which only change the speed.
• Hu et al. (2025), REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Normalization. The case for batch-level rather than per-prompt normalization.
• Shao et al. (2024), DeepSeekMath. Where GRPO started; its section on a “unified paradigm” of SFT, rejection sampling, DPO, PPO and GRPO as different gradient coefficients is an early version of this chapter’s master formula.

Next, Chapter 17 asks what to do when one reward at the end of a long response is not enough: rewarding individual steps, and spending more compute at test time instead of training time.