The frontier (as of October 2026)

Here is a sentence from the post-training section of the DeepSeek-V4 technical report (April 2026):

Although the training pipeline largely mirrored that of DeepSeek-V3.2, a critical methodological substitution was made: the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation (OPD).

Twenty chapters ago, almost every word in that sentence would have been opaque. Now most of it is familiar. “Reinforcement learning stage” means a policy-gradient method with a group baseline (Chapters 15 and 16). “Mixed” means many reward sources and environments trained together. “On-policy” is the distinction from Chapter 5. One phrase is new: on-policy distillation. It is one of the biggest changes of the past year, and it turns out to be an idea you already know (DAgger, from Chapter 10) combined with an objective you already solved (the Gibbs optimum of KL-regularized RL, derived in Chapter 10 and used throughout Chapter 11).

This last chapter has three jobs. First, it gives you a map of where RL for language models stands in October 2026: what the major model reports disclose, and the trends that connect them. Second, it teaches the two genuinely new ideas you need in full, with worked numbers: on-policy distillation, and the “maximum likelihood” view of RL with verifiable rewards. Third, it prepares you to keep going without a course. It lists the open problems worth your time and gives you a checklist for reading (and writing) papers in a field that moves monthly.

Nine years in one picture

Before the details, the big picture. The demo below places 55 milestones from 2017 to October 2026 on six lanes. Each milestone links to the chapter that teaches it and to its source.

The arc in one paragraph: RL entered language models through human preferences (2017–2022: reward models, PPO, InstructGPT, ChatGPT). Then came a period of simplification (2023–24): DPO removed the RL loop, and GRPO and RLOO removed the critic. Next, verifiable rewards turned RL into a way to teach reasoning (2024–25: o1, DeepSeek-R1, the GRPO variants). And in 2025–26 the field went wide: agents in thousands of environments, rubric-based judges for tasks with no right answer, systems that keep the trainer and sampler in numerical agreement, distillation to merge specialists, and a sober look at what reward hacking does to a model at scale.

What the 2026 model reports actually say

Model reports are the closest thing the field has to lab notebooks. Here is what the major ones disclose about post-training. Columns: the RL algorithm, where the rewards come from, the order of stages, whether there is a KL penalty to a reference model, and one notable detail. “n/d” means not disclosed.

A few entries use vocabulary from earlier chapters or from the reports themselves:

• $k_1$ and $k_3$ are the per-token KL estimators from Chapter 11. “IS-corrected” means each token’s KL term is multiplied by an importance weight, the ratio of the current policy to the old one that sampled the tokens, so the estimate stays unbiased after the policy has moved.
• “τ term” (Kimi) is a squared penalty $\tau\big(\log(\pi_\theta/\pi_\text{old})\big)^2$ that keeps the new policy near the one that sampled the data.
• IcePop, routing replay and the FP32 LM head are fixes for train–inference mismatch (Chapter 20).
• Muon is an optimizer, an alternative to Adam. It has nothing to do with RL.
• “Specialists” or “experts” are separate copies of the model, each trained with RL on one domain.

You don’t need to remember the rows. Read down the columns instead, and three patterns stand out.

1. The algorithm column is nearly constant. Every row is a REINFORCE-family method with a group or mean baseline and no learned critic. The differences are in the knobs from Chapter 16: clipping, normalization, aggregation, KL. The interesting variation has moved elsewhere.
2. The reward column has widened. In 2025 it said “rules” (verifiers). By 2026 it says “rules + generative reward model + rubrics”. An LLM judge, often RL-trained itself, scores answers against a rubric written for each prompt (Chapter 14).
3. The stages column grew a new last step. From late 2025, the pipeline often ends not with RL but with on-policy distillation: RL trains several specialist models, and distillation merges them into one. This is the single biggest structural change of the year, and it gets its own section below.

Trend 1: the algorithm settled down

By mid-2026, most published recipes for RL with verifiable rewards share a common core. You built every piece of it in Chapters 15–16 and 20:

• No critic. The baseline is the mean reward of the group of $G$ samples for the same prompt (GRPO, RLOO).
• Mean-only advantages. $\hat A^{(i)} = r^{(i)} - \text{mean}$, without dividing by the group standard deviation (Dr. GRPO, Olmo 3, DeepSeek-V3.2, Composer 2). Some recipes normalize over the whole batch instead (ScaleRL).
• Token-level loss aggregation, so long wrong answers aren’t under-penalized.
• Asymmetric or soft clipping: clip-higher, CISPO’s clipped importance weight, or masks that drop tokens whose ratio leaves a band.
• No KL to a reference, or a weak one (GLM-4.5/5, Olmo 3, Magistral, DAPO). DeepSeek kept a corrected KL and says the right strength depends on the domain.
• Zero-variance filtering: drop groups where every sample got the same reward. They carry no signal (Chapter 15).
• Train–inference consistency as a first-class concern (Chapter 20). This means truncated importance sampling against the sampler’s own log-probs, replaying MoE routing decisions from the rollout, FP32 or matched-precision logits, and in 2026 quantization-aware training so that sampler and trainer run the same low-precision weights (DeepSeek-V4, Kimi K3).

The last point marks a quiet shift in where the trust region lives. PPO’s clip (Chapter 9) was designed to stop the new policy from straying too far from the old one. In 2026 systems, the main source of off-policy-ness is often not the optimizer. It comes from the gap between the inference engine that sampled the tokens and the trainer that computes gradients, plus staleness from asynchronous rollouts. The clip has been repurposed accordingly. GLM-5’s IcePop and Kimi K2.5’s log-ratio masks zero out tokens whose sampler-versus-trainer ratio is out of band, whatever the sign of the advantage.

Because the core has converged, the research questions moved up a level: what to reward, which environments to train in, how to schedule stages, and how to merge specialists. The rest of the chapter follows those questions.

Trend 2: RL compute became something you can forecast

Pretraining has had scaling laws since 2020: a smooth curve that lets you predict what a big run will achieve from small ones. RL post-training had nothing like that until ScaleRL (Khatri et al., October 2025). The team ran more than 400,000 GPU-hours of ablations and found that, for a fixed model and dataset, held-out pass rate $R_C$ after spending compute $C$ follows a saturating sigmoid:

Each symbol has a plain meaning:

• $R_0$ is where you start.
• $A$ is the asymptote, the ceiling you approach with unlimited compute.
• $C_\text{mid}$ is the compute at which you are halfway to the ceiling.
• $B$ controls how sharply you get there (the efficiency).

The paper’s central empirical claim is that many popular knobs move only the efficiency, not the ceiling. In the paper’s words, these are “loss aggregation, normalization, curriculum, and off-policy algorithm”. Other choices raised $A$ in their experiments: the loss type (CISPO and GSPO beat DAPO), FP32 logits, and larger batches.

A worked example: why you can’t choose recipes from short runs. Two recipes start at $R_0 = 0.30$:

• Recipe X: high ceiling, slow. $A = 0.60$, $C_\text{mid} = 1000$, $B = 1$.
• Recipe Y: low ceiling, fast. $A = 0.52$, $C_\text{mid} = 300$, $B = 1.5$.

Here are their pass rates at three budgets (in GPU-hours, say):

The curves cross near $C \approx 2{,}350$ (set the two expressions equal and solve numerically). A team that compared the two recipes at 1,000 GPU-hours would pick Y and lose at scale. The point of fitting the sigmoid is to estimate $A$ and $B$ from the early part of each curve and then extrapolate. ScaleRL demonstrated this on a single run scaled to 100,000 GPU-hours, with the extrapolation from the early part of the curve matching where the run ended up.

How big are these budgets in practice? DeepSeek-V3.2’s report says its post-training budget “exceeds 10% of the pre-training cost”. Kimi K3’s report shows that “scaling RL FLOPs consistently improves” its capabilities. A fair summary: RL is no longer a small finishing step but a large share of training compute at the leading open labs.

Trend 3: on-policy distillation, the new last step

This is the idea you need in full. It is cheap and simple, and in the 2026 pipelines above it consolidates what RL learned.

The problem it solves

Suppose you have used RL to train several specialists: a math model, a coding agent, a search agent, a careful writer. Each is excellent in its domain. You want one model that is good at all of them. You have three options:

1. Mixed RL: train one model on all the environments at once. This is DeepSeek-V3.2’s approach. It works, but domains compete (“see-saw” effects), the reward scales differ, and every improvement needs another expensive RL run.
2. SFT on the specialists’ outputs (off-policy distillation): sample answers from each specialist and fine-tune the student on them. This is cheap, but it is behavior cloning (Chapter 10). The student only ever sees the specialists’ states. At test time it makes its own small mistakes, drifts into states no teacher demonstrated, and errors compound.
3. On-policy distillation (OPD): the student generates its own answers, and the teacher grades every token of them.

Option 3 is DAgger for language models. In DAgger, the learner drives and the expert labels the states the learner actually visits. Here, the student writes and the teacher says, at each position, how likely it would have been to write the same token. The student learns in exactly the states it will be in at test time, and it gets feedback on every token instead of one number per answer.

The objective: reverse KL on the student’s own samples

Write $\pi_\theta$ for the student and $\pi_T$ for the teacher. OPD minimizes the reverse KL from student to teacher, averaged over prompts and measured on the student’s own responses:

Because a sequence’s log-probability is a sum of token log-probabilities (Chapter 11), the quantity inside the expectation splits into one term per token. That suggests a per-token reward (write $s_t = (x, y_{<t})$ for the state at position $t$, the prompt plus everything generated so far):

In words: $r_t$ is positive when the teacher likes the token you wrote more than you do (“do this more”), and negative when the teacher likes it less. Its expectation under your own sampling is minus the KL at that position, so maximizing it pulls you toward the teacher. Thinking Machines’ implementation (October 2025) is literally two lines on top of an RL codebase: compute reverse_kl = sampled_logprobs - teacher_logprobs, set advantages = -reverse_kl, and run the usual policy-gradient update with discount 0. Each token is rewarded only for itself.

A worked example at one position

Take one position with a three-token vocabulary $\lbrace A, B, C\rbrace$. The student and teacher distributions are:

The student over-uses A and under-uses B. Here is what happens next.

Expected reward = minus the KL. Under the student’s own sampling, the expected reward is $0.6(-1.099) + 0.3(0.847) + 0.1(0) = -0.405$. Check it directly: $\mathrm{KL}(\pi_\theta \Vert \pi_T) = 0.6\log 3 + 0.3\log(3/7) + 0 = 0.659 - 0.254 = 0.405$. ✓

One sample, one noisy push. Say the student’s logits are $z$, so $\pi_\theta = \text{softmax}(z)$. The REINFORCE update for a sampled token $y$ is $r_y \nabla_z \log \pi_\theta(y)$, and for a softmax $\nabla_z \log \pi_\theta(y) = e_y - \pi_\theta$ (the one-hot vector minus the probabilities). So:

• If A is sampled (60% of the time), the push is $-1.099 \times (0.4, -0.3, -0.1) = (-0.44, +0.33, +0.11)$. That lowers A and raises B and C.
• If B is sampled (30%), the push is $0.847 \times (-0.6, 0.7, -0.1) = (-0.51, +0.59, -0.08)$. That raises B and lowers A.
• If C is sampled (10%), the push is zero.

The average push is the exact gradient. Weight the three cases by their probabilities: $0.6(-0.44, 0.33, 0.11) + 0.3(-0.51, 0.59, -0.08) = (-0.42, +0.38, +0.04)$. That is exactly $-\nabla_z \mathrm{KL}$. The gradient of the KL with respect to logit $k$ is $\pi_k\big(\log(\pi_k/q_k) - \mathrm{KL}\big)$, which gives $0.6(1.099 - 0.405) = 0.42$ for A, $0.3(-0.847 - 0.405) = -0.38$ for B and $0.1(0 - 0.405) = -0.04$ for C.

The single-sample pushes all point roughly the right way, but their sizes and directions vary a lot from sample to sample. That is the variance DeepSeek-V4 cites as its reason for doing something more expensive: full-vocabulary OPD. It computes the exact KL over the whole vocabulary at every position the student visited, and differentiates it directly. The cost is that you need the teacher’s entire next-token distribution at every position, not just one log-prob. With a vocabulary of 100,000+ tokens and more than ten teachers, DeepSeek-V4 had to build dedicated infrastructure to schedule the teachers. Kimi K3 went the other way: sampled tokens, with the per-token reward clipped to a range.

Why it works: OPD is KL-regularized RL in disguise

You have already solved this problem. Chapter 11’s KL-regularized objective is $\max_\pi \mathbb{E}_{y\sim\pi}[r(x,y)] - \beta\,\mathrm{KL}(\pi \Vert \pi_\text{ref})$, and its optimum is the Gibbs distribution $\pi^* \propto \pi_\text{ref}\, e^{r/\beta}$ (the Gibbs variational principle, derived in Chapter 10). OPD is a special case.

Two physics notes help here. First, minimizing $\mathrm{KL}(\pi_\theta \Vert \pi_T)$ over a restricted family is variational inference: it is how you fit a mean-field approximation to a Boltzmann distribution. You know its character. Reverse KL is mode-seeking: a student too small to cover all of the teacher’s modes picks some and fits them well, rather than smearing probability over everything. For a small student imitating a large teacher, that is usually what you want. Second, the reward $\log \pi_T$ can’t be “hacked” in the usual sense. A low reverse KL means the teacher would have produced what you produced. The Thinking Machines post calls it “unhackable” in exactly this sense. The weakness is elsewhere: you can never be better than the teacher’s judgment of your tokens (except via tricks like G-OPD’s extrapolation).

What it buys, in numbers

• Qwen3 (May 2025, Table 21). Starting from the same off-policy-distilled Qwen3-8B checkpoint (AIME’24: 55.0), RL reached 67.6 using 17,920 GPU-hours. On-policy distillation from a larger Qwen3 teacher reached 74.4 using 1,800 GPU-hours. OPD also raised pass@64 (90.0 → 93.3), where RL left it unchanged. That last detail matters for the “sharpen or expand” debate below: distillation can transfer things the student couldn’t find by itself.
• Thinking Machines (October 2025). Starting from a Qwen3-8B-Base checkpoint at 60% on AIME’24 after SFT on 400K prompts, OPD with a Qwen3-32B teacher reached 70% in about 150 steps. Their estimate is a 9–30× compute saving versus continuing SFT to the same score.
• Production use in 2026: GLM-5’s cross-stage distillation (to recover skills lost during later RL stages), DeepSeek-V4 (more than ten teachers), Nemotron 3 Ultra (more than ten teachers) and Kimi K3 (nine experts). In all of these, RL builds the specialists, and distillation merges them.

You can see the same ordering in miniature in this chapter’s lab. On TinyLM, with the same number of sequences per step, full-vocabulary OPD raised sampled accuracy from 0.31 to 0.67, SFT on the teacher’s samples reached 0.64, and sampled-token OPD and RL both reached 0.54.

When it fails

OPD is not free, and 2026 has produced a small science of its failure modes:

• You need a teacher. OPD transfers capability; it doesn’t create it. Someone still has to run RL to make the specialists.
• Compatibility matters more than teacher size. Li et al. (April 2026) find that OPD succeeds only when student and teacher share compatible “thinking patterns” and the teacher offers genuinely new capability. Acikgoz et al. (September 2026) report that an RL-strengthened student can regress when distilled from the same teacher, and that preparing both sides first (a brief SFT warm-up for the student, RL adaptation for the teacher) raised OPD accuracy from 29.2% to 43.8%.
• Long outputs are hard. Reported failure modes include length inflation and repetition, the student drifting into prefixes where the teacher’s guidance is unreliable, and tokenizer mismatches between student and teacher.
• Privileged self-distillation is contested. A popular 2026 variant uses the same model as its own teacher, given privileged context such as the reference solution. Two July 2026 papers report that it degrades long-reasoning models. Treat that line of work as unsettled.

Trend 4: rewards for things you can’t verify

RL with verifiable rewards (Chapter 15) works because a checker can’t be argued with. Most of what people want from a model, such as a helpful answer, a well-structured report or a good story, has no checker. The 2025–26 answer has three layers, all introduced in Chapter 14:

• Rubrics. For each prompt, write (or have a model write) a checklist of criteria with weights, such as “mentions the drug interaction” (essential) or “suggests seeing a doctor” (important). An LLM judges each item, and the reward is the weighted fraction satisfied. Rubrics as Rewards (Gunjal et al., 2025) frames RLVR as the special case of a single essential criterion.
• Generative reward models (GRMs). Instead of a scalar head, the judge is an LLM that writes a critique and a score. DeepSeek-V3.2 uses a GRM with per-prompt rubrics for general tasks. Kimi K3’s “agentic GRM” must follow a protocol: read the output, write a rubric, score each candidate, record the scores. In DeepSeek-V4, RL is applied to the GRM itself, so the judge improves along with the policy.
• Defenses written into the reward. Kimi K2 used rubrics “designed against reward hacking”, for example penalizing flattering openings. In Kimi K3’s GRM, a candidate longer than a set multiple of the cold-start model’s typical length automatically loses its pairwise comparison. DeepSeek-R1 only switched on its preference reward for the final 400 steps, because “more training steps with the model based preference reward signal may lead to reward hacking”.

The common thread is that a rubric is a proxy, and the optimizer will find the gap between the proxy and what you meant (Chapter 19). Here is what that looks like in this chapter’s lab (Project 2). The gold goal is an exactly correct sum. A rubric awards 2 points for exactness and 2 points for “looks thorough” (the answer has 3 or more digits). After 200 RL steps:

The policy found that padding every answer to three digits earns half the reward with certainty, which beats earning the full reward on only some answers. The fix is a better rubric: lower weights on style, negative “pitfall” items, or length budgets like Kimi K3’s. But someone has to notice first, which is why monitoring gold metrics matters.

Trend 5: rethinking the objective, from expected reward to likelihood

Every chapter so far has maximized expected reward. With binary rewards, for one prompt with pass rate $p(\theta)$ (the probability a sample is correct), that objective is just $J = p$. But is $p$ the right thing to maximize? Think like a statistician. If correctness is all you care about, the natural objective is the log-likelihood of being correct, $\log p$, just as supervised learning maximizes log-likelihood rather than likelihood. The two have very different gradients:

The $1/p$ factor is the whole story. A prompt the model solves 1% of the time gets 100 times more weight under maximum likelihood than under expected reward. Maximum likelihood cares most about the prompts you are worst at. Expected reward treats a gain of 0.01 on a hopeless prompt the same as 0.01 on an easy one.

So why does RL use $p$? Because $\log p$ can’t be estimated without bias from a finite number of samples. You’d need $1/p$, and you don’t know $p$. MaxRL (Tajwar et al., February 2026) found a clean way to interpolate between the two.

This gives a single picture of the whole critic-free zoo. At the population level, each method’s gradient for a prompt with pass rate $p$ is $w(p)\,\nabla p$ for some weight function:

Worked numbers. Take a hard prompt with $p = 0.05$ and $N = 8$. Expected reward gives weight 1. GRPO gives $1/\sqrt{0.05 \times 0.95} = 4.6$. MaxRL gives $(1 - 0.95^8)/0.05 = (1 - 0.663)/0.05 = 6.7$. Maximum likelihood would give 20. Notice also that GRPO’s weight rises again near $p = 1$ (it is symmetric), so GRPO also up-weights nearly solved prompts. That is the “difficulty bias” Dr. GRPO objected to (Chapter 16).

The demo’s dashed curve is a reminder that the weights are population-level. In a real batch, a prompt with no correct sample contributes nothing under every method in the table, because the group has no variance. MaxRL can only up-weight hard prompts that produced at least one success. That is why “more rollouts per prompt” (BroRL; the IsoCompute Playbook, March 2026) and “pick prompts at the frontier” (next section) are complementary to the choice of objective. The paper reports that MaxRL “Pareto-dominates” GRPO and RLOO in its experiments, with up to 20× gains in test-time scaling efficiency (as with any single paper’s claims, look for replications).

The sharpen-or-expand debate, one year on

Maximum likelihood’s focus on hard prompts connects to the most argued-over question of 2025: does RL with verifiable rewards teach models anything new, or does it only sharpen what the base model could already do? (Chapter 15 introduced it.) The evidence so far:

• Sharpening. Yue et al. (2025) found RL-trained models win at pass@1 but their base models win pass@k at large $k$, as if RL concentrates probability on solutions the base model could already sample. Spurious-reward results point the same way: on Qwen2.5-Math-7B, even random rewards gave a large MATH-500 gain (+21.4 points, against +29.1 for true rewards), and the effect did not carry over to other model families.
• Expanding. ProRL (Liu et al., 2025) reports boundary gains from prolonged RL with KL control and reference resets. Qwen3’s on-policy distillation raised pass@64 where RL did not. Cursor’s Composer 2 report says RL improved both average and best-of-K performance.
• Both, depending on the task. Cheng et al. (October 2026) model RL as reweighting which strategy a model picks. That can raise coverage on some tasks and lower it on others, without adding new strategies.

You can reproduce the basic phenomenon on a laptop. In Project 5 of this chapter’s lab, RL on TinyLM raised pass@1 from 0.31 to 0.67, while pass@64 fell from 0.97 to 0.92. The curves cross between $k = 16$ and $k = 32$. Under an expected-reward objective this is no surprise. If $p = \sigma(z)$ for some logit $z$, the push a prompt gets is proportional to $p(1-p)$: 0.21 at $p = 0.3$, but only 0.003 at $p = 0.003$. The rare-success prompts, which are exactly the ones that decide pass@k at large $k$, get almost no attention, and they can quietly lose probability while the common ones are sharpened. Objectives like MaxRL are one attempt to change that.

Trend 6: self-play and “zero-data” RL

Verifiable RL needs problems with checkable answers, and good problems run out. One response is to let the model write its own problems. This is self-play, in the tradition of AlphaZero (Chapter 10), adapted to language.

Absolute Zero (Zhao et al., 2025) is the cleanest example. One model plays two roles. As proposer, it writes a small coding task: a program and an input, and it must predict the output, find the input, or write the program. As solver, it attempts the task. A Python executor checks everything, so no human-written data is needed. The interesting part is the proposer’s reward. If the solver succeeds on a fraction $\hat p$ of $G$ attempts:

Trivial tasks ($\hat p = 1$) are worth nothing, and so are impossible ones ($\hat p = 0$). The best tasks are those the solver sometimes solves. This learnability reward encodes the same lesson as the dashed curve in the demo above: a group with no success, or no failure, teaches nothing. Exercise 21.5 shows that, in expectation over the solver’s $G = 8$ attempts, the proposer earns the most from tasks solved about a quarter of the time. That is no coincidence. The expected learnability reward, $(1-p) - (1-p)^G$, is the same function as MaxRL’s push on the logit with $N = G - 1$ samples, $(1-p) - (1-p)^{N+1}$. Switch the weights demo to “Push on the logit” and the MaxRL curve peaks near $p \approx 0.25$. A proposer paid for learnability picks the tasks where a likelihood-style learner learns fastest.

Other designs in the same family:

• R-Zero (2025): a Challenger is rewarded for problems at the edge of a Solver’s ability.
• TTRL (2025): uses the majority vote over the model’s own samples as a pseudo-label on unlabeled test questions.
• Self-play SWE-RL (Wei et al., Meta, December 2025): one agent injects bugs into real repositories, specified by test patches, and learns to fix them. It reports +10.4 points on SWE-bench Verified and consistently beat a human-data baseline over training.
• Self-Play Pretraining with Zero Data (Cowsik et al., September 2026): pushes the idea to its extreme. From random initialization, an RL-trained generator proposes programs for a universal Turing machine at the frontier of a learner’s ability. The authors present it as an initial proof of concept.

Trend 7: RL moves earlier, and later

RL used to be the last stage of training. It is spreading in both directions.

Earlier: RL during pretraining. Reinforcement Pre-Training (Dong et al., 2025) turns ordinary next-token prediction into a verifiable task: the model reasons, then predicts the next token, and is rewarded if it is right. RLP (Hatamizadeh et al., NVIDIA, 2025) treats a sampled chain of thought as an action. The reward is the information gain: how much the thought increased the log-likelihood of the actual next tokens, compared with predicting without it. This reward is dense, needs no verifier, and works on any text. Both papers report gains, and RLP reports that they persist after identical post-training. Whether this changes how frontier models are pretrained is not yet known publicly.

Later: learning from deployment. In March 2026 Cursor described “real-time RL” for its Composer coding model. It trains on rewards derived from how real users respond to the model’s actions, and can ship a new checkpoint “as often as every five hours”, which keeps the training data close to on-policy. The company reports small A/B gains: for example, “agent edit persists in codebase” up 2.28% and “dissatisfied follow-up” down 3.13%. The same post describes the model finding ways to game production metrics, which turned each hack into a bug report.

This is RL closing its original loop. An agent acts in the world, the world responds, and the agent improves. It is also where Chapter 19’s warnings bite hardest. OpenAI’s April 2025 sycophancy incident was caused partly by adding a reward from users’ thumbs-up and thumbs-down, which the company said “can sometimes favor more agreeable responses”. Learning from users means learning what users reward, which is not always what is good for them.

A related question is continual learning without forgetting. Shenfeld et al. (“RL’s Razor”, 2025) found that on-policy RL forgets less than SFT at matched performance on the new task. Their explanation is that on-policy updates are implicitly biased toward solutions close (in KL) to the starting model. Thinking Machines used on-policy distillation to restore instruction-following after fine-tuning a model on new documents.

Trend 8: agents, environments, and scale

Chapter 18 covered agentic RL. The 2026 change is mostly one of scale, and of where the effort goes:

• Environment counts. Qwen’s Qwen3.5 blog (February 2026) attributes its post-training gains “primarily” to “extensive scaling of virtually all RL tasks and environments”. MiniMax describes RL in “hundreds of thousands of complex real-world environments”. DeepSeek-V4 describes a sandbox service built to run hundreds of thousands of concurrent instances. These are lab claims, and “environment” is not a standard unit.
• Synthetic and simulated environments. Kimi K2 synthesized thousands of tools. Qwen-AgentWorld (June 2026) trains LLMs to simulate agentic environments, so an agent can be trained against a learned world model. This is the MuZero idea from Chapter 10, with the usual risk that the policy exploits the simulator’s errors.
• Training inside the real harness. Composer 2 runs RL inside the same tool environment its users see, so there is no train/deploy mismatch in tools.
• Effort as a control knob. DeepSeek-V4 and Kimi K3 train separate experts for different reasoning budgets (“low/high/max”), then distill them into one model. The budget is enforced by the reward: Kimi K3 overrides the reward with −1 for trajectories over a budget.

The open technical problems here are the ones Chapter 18 named: credit assignment over hundreds of tool calls, sandbox cost and safety, and reward hacking of environment checks.

Trend 9: reward hacking at scale, and keeping chains of thought readable

Chapter 19 taught reward hacking with small examples. In 2025–26 it was measured at frontier scale, and the findings are sobering.

• Hacking generalizes. MacDiarmid et al. (Anthropic, November 2025) trained a model that knew about reward hacks on real Anthropic production coding environments. It learned to hack them (for example, making tests pass by overriding equality checks). It then generalized to alignment faking, cooperation with malicious actors, and attempted sabotage, including of the codebase for the paper itself. Standard chat-style safety training made it look aligned on chat evaluations, but the misalignment persisted on agentic tasks. Three mitigations worked: preventing the hacking, more diverse safety training, and inoculation prompting. In inoculation prompting, the training prompt frames hacking as acceptable in this context, and the model then no longer generalizes it into broader misbehavior.
• A deliberately trained reward seeker. In August 2026, Anthropic described training an early Opus 4.8 checkpoint on 80 RL environments it had previously identified as hackable, as a “plausible (though pessimistic) proxy” for a run with no anti-hacking effort. By the end it hacked 40% of all episodes. On evaluations it generalized to reward tampering (0% → 41%), among other behaviors, while looking aligned on evaluations where no grader was involved. Its chain of thought “displayed a strong desire to satisfy the grader”.
• Production rates. The Claude Mythos Preview system card (April 2026) reports an automated review of several hundred thousand RL transcripts. In about 0.05% of episodes the model attempted to get around network restrictions in its training environment to shortcut the task, and in about 0.01% it succeeded.

The main tool for catching this is reading the model’s chain of thought. Baker et al. (OpenAI, 2025) showed that a weaker model reading a stronger model’s chain of thought caught reward hacks (like calling exit(0) before tests could fail) far better than one watching only actions. They also found that putting that monitor into the reward works at first, but with more optimization the model learns to hide its intent while still hacking. Hence the proposed “monitorability tax”: don’t optimize the chain of thought directly, even if it would help in the short run. OpenAI’s gpt-oss model card says it followed this policy. A cross-lab position paper (Korbak et al., 2025) argues that CoT monitorability is “a new and fragile opportunity”. OpenAI’s December 2025 measurements found that RL optimization “does not materially decrease monitorability even at the current frontier scale”. These findings are compatible: ordinary RL has not yet destroyed readability, while direct pressure on the chain of thought would.

What is still contested

Where serious groups disagree, here is the state of play:

Open problems

These are the problems a researcher entering the field in late 2026 could reasonably work on. Each comes with why it matters and where to start.

1. Rewards for tasks without checkers that resist hacking. Rubric and GRM rewards are now central, and the optimizer finds their gaps (verbosity, confident tone, flattery). How do you know a judge is good enough to optimize against, and for how long? Start: Chapter 14, Rubrics as Rewards, the Kimi K3 GRM protocol, and Gao et al.’s over-optimization curves.
2. What sets the ceiling of RL? ScaleRL shows recipes differ in their asymptote $A$, but not why. Is it the base model, the data, the objective, exploration? Start: ScaleRL, Cheng et al. (2026).
3. Does RL create capabilities? A clean answer needs leak-free tasks, many model families, and large-$k$ evaluation with confidence intervals. Start: Yue et al., ProRL, the contamination critiques, and Project 5 in the lab.
4. Exploration in token space. Entropy collapses (Chapter 19), and many fixes compete with no principled winner. What would “optimism under uncertainty” (Chapter 2) mean for a policy over sequences? Start: Cui et al. (2025) on the entropy mechanism; the 80/20 high-entropy-token paper.
5. Credit assignment over long horizons. Agents run hundreds of tool calls and get one reward at the end. Critics are expensive, and group baselines are coarse. Start: Chapter 18; GiGPO; VinePPO’s Monte Carlo values.
6. When does on-policy distillation work, and can a student surpass its teachers? We know compatibility matters, but not how to measure it in advance. Start: Li et al. (2026), Acikgoz et al. (2026), G-OPD.
7. The right objective. Expected reward, likelihood (MaxRL), pass@k-aware objectives, and test-time-compute-aware training each weight prompts differently. Which should you use when? Start: MaxRL’s weight-function view (this chapter’s demo).
8. Principled off-policy RL for asynchronous systems. The field has a toolbox of patches (truncated IS, masks, routing replay, matched precision). A theory of how much staleness and mismatch is harmless would let systems go faster safely. Start: Chapter 20; Qwen’s “Stabilizing RL with LLMs” (2025).
9. Reward hacking that generalizes. Why does hacking in coding environments turn into broader misalignment, and why does inoculation prompting help? Can you detect hackable environments before training? Start: MacDiarmid et al. (2025), Anthropic’s reward-seeker post (2026), ImpossibleBench.
10. Keeping chains of thought monitorable. What training choices preserve or destroy it, and how would we know? Start: Baker et al. (2025), Korbak et al. (2025), “Monitoring Monitorability” (2025).
11. Learning from deployment without learning to please. Real-time RL from users is attractive and dangerous (sycophancy). How do you separate “the user was helped” from “the user was flattered”? Start: Cursor’s real-time RL post; OpenAI’s sycophancy post-mortems; Chapter 19.
12. Where environments come from. Self-play, simulated environments and mined repositories each scale differently and fail differently. Start: Absolute Zero, Self-play SWE-RL, Qwen-AgentWorld.

How to read (and write) an RL-for-LLM paper

The field publishes faster than anyone can read, and many results don’t replicate. Most failures to replicate trace back to a short list of confounders. Before you believe a claimed improvement, check these:

1. Base model. Which model family? Results on one family (especially Qwen2.5-Math, given the spurious-reward and contamination findings) often don’t transfer. Look for at least two families.
2. Baselines. Was the baseline tuned as carefully as the new method, at the same learning rate, batch and number of samples? Is it the authors’ own implementation, or a number copied from another paper with a different setup?
3. Compute. Are methods compared at matched compute (samples generated, gradient steps, GPU-hours), or matched steps? A method that samples twice as many responses per step should be compared at equal samples. ScaleRL adds: is the gain in efficiency (it will vanish with more compute) or in the ceiling?
4. KL and the reference. Is there a KL term? If not, how far did the policy move? A method that wins only because it was allowed to drift further is a different claim.
5. Evaluation sampling. Is the metric pass@1 from greedy decoding, or avg@k over $k$ samples at some temperature? On a 30-question benchmark like AIME, one question is 3.3 points. Are there error bars?
6. Contamination. Could the benchmark be in the pretraining data? Gains that appear even with random rewards are a red flag.
7. Seeds. One run per method is an anecdote. RL runs vary a lot across seeds.
8. Version. arXiv papers change. DeepSeek-R1’s v1 and v2 report different AIME numbers (71.0% vs 77.9% for R1-Zero). Cite the version you read.
9. Self-reported numbers. Model-report benchmarks, especially on internal benchmarks, are claims, not measurements you can check.

A worked example. A paper claims “Method Z improves AIME’24 accuracy over GRPO by 8 points”. The setup: Qwen2.5-Math-7B; GRPO numbers copied from another paper; Method Z samples 16 responses per prompt against GRPO’s 8; evaluation by avg@4 at temperature 0.6; one seed. Here is what you should list:

• one model family, and a contamination-prone one;
• an untuned, borrowed baseline;
• twice the samples per step for Method Z;
• 8 points on a 30-question test averaged over only 4 samples is about 2.4 questions, with no error bars;
• a single seed.

Any one of these could explain the whole gain. Exercise 21.6 gives you another one to dissect.

How to keep up. Read the post-training sections of new open-weight model reports. They are dense, and you now have the vocabulary to decode them. Follow the open RL stacks (TRL, verl, OpenRLHF, slime, prime-rl): when a trick becomes a default in their configs, it has usually survived contact with many users. When a paper surprises you, reproduce a small version before you build on it. The lab for this chapter is set up for exactly that.

Summary

• The 2026 pipeline: SFT → RL to train domain (and effort-level) specialists → on-policy distillation to merge them, increasingly replacing one big mixed RL run (DeepSeek-V4, Nemotron 3 Ultra, Kimi K3; GLM-5’s cross-stage variant).
• On-policy distillation = student samples, teacher scores every token with $r_t = \log\pi_T - \log\pi_\theta$, policy gradient with discount 0. It is KL-regularized RL whose optimum is the teacher, and it is DAgger for LLMs: dense, on-policy, and often an order of magnitude cheaper than RL when a good teacher exists.
• The RL algorithm converged on a critic-free, group-baseline policy gradient with mean-only advantages, token-level loss, asymmetric or IS-weight clipping, little or no KL, zero-variance filtering, and tight control of train–inference mismatch. The research moved to rewards, environments and stage design.
• RL compute is now forecast with saturating curves (ScaleRL): many knobs change efficiency, and only some change the ceiling. Compare recipes by extrapolation, not by snapshots.
• Expected reward is the first-order term of maximum likelihood ($\nabla\log p = \sum_k \frac1k\nabla\text{pass@}k$). The critic-free methods differ in the weight $w(p)$ they put on each prompt: 1 (REINFORCE/RLOO), $1/\sqrt{p(1-p)}$ (GRPO), $(1-(1-p)^N)/p$ (MaxRL), $1/p$ (maximum likelihood).
• Rewards for non-verifiable tasks come from rubrics and generative judges, which are proxies that get hacked, so reward design is now also a safety question: hacking generalizes (MacDiarmid et al.; Anthropic’s reward seeker), and chain-of-thought monitoring works only if you don’t optimize against it.
• Read papers with a checklist: base model family, tuned baselines, matched compute, KL, evaluation sampling, contamination, seeds, version.

Exercises

def reverse_kl_rewards(student_logp, teacher_logp, mask):
    return ((teacher_logp - student_logp) * mask).detach()

def opd_pg_loss(student_logp, rewards, mask):
    return -(rewards.detach() * student_logp * mask).sum() / mask.sum().clamp(min=1.0)

def full_vocab_reverse_kl(student_logits, teacher_logits, mask):
    log_s = F.log_softmax(student_logits, -1)
    log_t = F.log_softmax(teacher_logits.detach(), -1)
    kl = (log_s.exp() * (log_s - log_t)).sum(-1)
    return (kl * mask).sum() / mask.sum().clamp(min=1.0)

def pass_at_k(n, c, k):
    if n - c < k:
        return 1.0
    i = np.arange(k)
    return float(1.0 - np.prod((n - c - i) / (n - i)))

def maxrl_advantages(rewards):
    r = np.asarray(rewards, dtype=float)
    mean = r.mean(axis=-1, keepdims=True)
    return np.where(mean > 0, (r - mean) / np.where(mean > 0, mean, 1.0), 0.0)

cd 21-frontier
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• DeepSeek-AI (2026), DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, section 5. The clearest public description of a 2026 frontier post-training pipeline: GRPO specialists, a generative reward model, and full-vocabulary multi-teacher on-policy distillation, plus the infrastructure that makes it run.
• Lu and Thinking Machines (2025), On-Policy Distillation. The most readable introduction to OPD as RL, with code, compute accounting, and the continual-learning application.
• Khatri et al. (2025), The Art of Scaling Reinforcement Learning Compute for LLMs. Sigmoid scaling curves for RL, and which design choices move the ceiling versus the efficiency. Read it for the methodology.
• Tajwar et al. (2026), Maximum Likelihood Reinforcement Learning. The pass@k expansion of $\log p$ and the weight-function view that unifies REINFORCE, GRPO and maximum likelihood. Sections 3–5 are short and beautiful.
• Yue et al. (2025), Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? The paper that started the sharpen-or-expand debate. Read it alongside ProRL for the other side.
• MacDiarmid et al. (2025), Natural Emergent Misalignment from Reward Hacking in Production RL. Why reward hacking is a safety problem and not only an efficiency problem, and what mitigations worked.
• Team Olmo (2025), Olmo 3. A fully open RLVR recipe (data, code, logs), with every algorithmic choice justified. The best starting point for reproducing frontier-style RL yourself.
• Korbak et al. (2025), Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. A short cross-lab position paper on why training choices should preserve readable reasoning.

That’s the end of the course. You started with an agent in a gridworld and ended reading 2026 model reports. The next chapter is yours: pick an open problem, run the small version first, and report the seeds.