Agentic and multi-turn RL

A test in a small Python repository is failing, and you ask a coding agent to fix it. The agent reads add.py, runs the tests, sees add(2, 3) returned -1, expected 5, changes a - b to a + b, runs the tests again, sees 3 passed, and submits. That took four turns. A good part of the text in the agent’s context window was written by the tools, not the model. And the training signal at the end is a single bit: did the held-out tests pass?

In Chapter 15 the model read a prompt, wrote one response, and got a reward. That is a one-step problem, a contextual bandit with an enormous set of arms. This chapter is about what changes when the model acts: it calls tools, reads what comes back, and decides what to do next, over many turns. Three questions come up that a single response never raised:

1. Which tokens do you train on? The context now contains text the model didn’t write.
2. Which decision gets the credit? One reward has to be shared among many turns.
3. How do you run this at all? Episodes are long, slow, need sandboxes, and can be gamed in new ways.

The answers draw directly on Part II. Question 1 is the policy gradient theorem from Chapter 7, where the environment’s dynamics drop out of the gradient. Question 2 is credit assignment from Chapter 8. Question 3 leads to the systems of Chapter 20 and the failure modes of Chapter 19.

From one answer to many turns

The agent loop

Here is the episode above written out as the model sees it. The model’s own text is between <call> tags and the tool’s replies are between <result> tags. Every LLM agent framework uses some format like this, though the exact tags differ.

[prompt]  system: Tools: read(path) run_tests() edit(path, old, new) submit()
          user: test_add fails. Fix add.py so the tests pass.
[model]   <think> Look at the code first. </think> <call> read("add.py") </call>
[tool]    <result> def add(a, b): return a - b </result>
[model]   <think> Minus should be plus. </think> <call> edit("add.py", "a - b", "a + b") </call>
[tool]    <result> ok: 1 line changed </result>
[model]   <call> run_tests() </call>
[tool]    <result> 3 passed </result>
[model]   <call> submit() </call>
          -> grader runs the held-out tests: pass. Reward 1.

Some vocabulary, used throughout the chapter:

• A tool is any function the model can invoke by writing text in an agreed format: a code interpreter, a search engine, a file editor, a shell.
• A tool call is the model’s text that requests one, and the observation (or tool output) is the text that comes back.
• A turn is one model message plus the observation that answers it. The episode above has four turns.
• The harness (also called the scaffold) is the program around the model. It parses tool calls, executes them, appends the results to the context, enforces limits (turns, tokens, time), and calls the grader at the end.
• The environment is everything the model doesn’t control: the harness, the tools, the repository or website they act on, and the grader.
• An agent is the model together with this loop. Agentic RL means training the model with RL inside the loop, on whole episodes, rather than on single responses.

What actually changes

Compared with Chapter 15, three things are new.

The environment writes into the context. In the single-response setting, every token after the prompt was sampled from the policy. Now the context is a braid of model tokens and environment tokens. In the example above the tool’s replies are short, a few tokens each. In real coding agents the tools often write most of the context: a single file read or test log can be thousands of tokens.

Rewards arrive after many decisions. The agent made four decisions, then got one number. In a software-engineering task the agent may take dozens or hundreds of turns. Kimi reports that K2 Thinking can execute “200 – 300 sequential tool calls” (Moonshot, 2025).

Episodes are expensive. Every tool call takes wall-clock time, may need a sandboxed container, and can fail or hang. A rollout is no longer “generate 2,000 tokens”; it is a small distributed program.

We take these in order: first the formalism and the loss mask, then rewards and credit, then environments and the practical problems.

The multi-turn MDP

Chapter 11 described generation as a token-level MDP: the state is the prompt plus the tokens so far, the action is the next token, and the transition is deterministic (append the token). Agents need one change: at certain moments the environment appends tokens, and those depend on the world, not on the model.

Write a $K$-turn episode as

where $x$ is the prompt, $a_k$ is the model’s message at turn $k$ (a sequence of tokens), and $o_k$ is the observation the environment returns. The history before turn $k$ is everything in the context so far, $h_k = (x, a_0, o_0, \dots, a_{k-1}, o_{k-1})$. The model writes $a_k$ one token at a time from $\pi_\theta(\cdot \mid h_k, \text{tokens of } a_k \text{ so far})$, so

where $t \in a_k$ means “the positions of $a_k$‘s tokens”. Then the environment draws $o_k \sim P(\cdot \mid h_k, a_k)$. Rewards $r_k$ may come after any turn, but often only the last one is nonzero.

You can view this at two levels:

Both are useful. The gradient is computed at the token level, because that is how the model factorizes probability. Credit is often easier to reason about at the turn level, because that is where the environment responds and where rewards arrive.

Which tokens do you train on? The loss mask

The environment’s tokens drop out of the gradient

Start from the probability of a whole episode under the policy. The prompt comes from the dataset, the model writes each message, and the environment writes each observation:

(The last message $a_{K-1}$ gets no observation; the episode ends.) Take the logarithm and the gradient. The prompt distribution $\mathcal{D}$ and the environment $P$ don’t depend on $\theta$, so their terms vanish:

This is the step from Chapter 7 where the transition model dropped out of the policy gradient. There it explained why REINFORCE works without knowing the physics of CartPole. Here it says something concrete about tokens: the policy gradient is a sum over the tokens the model wrote, and only those. The tool’s tokens are part of the state the policy conditions on; they are not actions. With a baseline and per-turn advantages $\hat{A}_k$ (next section), the estimator is

In code, you don’t write separate loops over turns. You lay the whole episode out as one token sequence, run the model over it once (exactly as in Chapter 15), and multiply the per-token loss by a loss mask $m_t$ that is 1 for tokens the model sampled and 0 for everything else (prompt, tool output, padding).

A worked example

Here is one episode from this chapter’s lab. A secret number in $0..7$ is hidden (?). The agent can call compare(x), written call x, and the tool answers lower, higher or equal. Each model turn is two tokens.

position    0       1       2      3      4       5       6      7      8       9       10     11
token       <bos>   ?       call   3      <tool>  higher  call   5      <tool>  lower   ans    4
written by  prompt  prompt  model  model  tool    tool    model  model  tool    tool    model  model
mask m_t    0       0       1      1      0       0       1      1      0       0       1      1
turn        -       -       0      0      0       0       1      1      1       1       2      2

The episode ends with the correct answer, so the agent earns 1, minus 0.05 for each of its two tool calls. Six of the twelve tokens get gradient. The model still reads all twelve: the hidden state after higher is what lets it choose 5 next. Reading and training are different things. The model conditions on the tool tokens but is never asked to have predicted them.

In PyTorch, the mask adds almost nothing to Chapter 15’s loss:

logp = log_softmax(policy(ids[:, :-1]), -1).gather(-1, ids[:, 1:, None]).squeeze(-1)
loss = -(mask[:, 1:] * adv[:, 1:] * logp).sum() / mask[:, 1:].sum()

The same mask must be applied everywhere a per-token quantity is summed: the KL penalty, any entropy bonus, and the token count in the normalizer. Search-R1 states that its retrieved-token masking “is also applied when calculating the KL divergence loss” (Jin et al., 2025).

What goes wrong without the mask

Suppose you forget the mask and train on every token after the prompt. Three things go wrong at once.

It is no longer the policy gradient. The extra terms are $\hat{A}_k \nabla_\theta \log \pi_\theta(o_{k,t} \mid \cdot)$ for each tool token. They reward the model for having predicted the tool’s output in good episodes and penalize it in bad ones. That is advantage-weighted imitation of the environment. On a good episode it teaches the model to write 3 passed itself after an edit, which is exactly how an agent learns to hallucinate tool results. Worse, a baseline is no longer free. In Chapter 7, subtracting a baseline $b$ cost nothing because $\mathbb{E}_{y \sim \pi_\theta}[\nabla \log \pi_\theta(y)] = 0$. Tool tokens are drawn from the environment, not from $\pi_\theta$, so that expectation is not zero and the baseline term adds a push of its own (Exercise 18.2).

There is no self-correction. When the model’s own token gets pushed down, it becomes less likely, gets sampled less often, and so stops being pushed. Tool tokens are forced into the context regardless of their probability. The gradient of $\log \pi_\theta(o)$ with respect to the logits is $e_o - \pi_\theta$ (here $e_o$ is the one-hot vector for token $o$). Its size is at least $1 - \pi_\theta(o)$, so it does not shrink as $\pi_\theta(o) \to 0$. A tool token on a bad episode gets shoved down again every time it appears, without limit.

The extra terms are large. Tool output is often most of the episode, and it is text the policy never chose, so it finds those tokens improbable. In the lab, at the start of training, tool tokens are 40% of the episode tokens and the policy gives them an average log-probability of −7.8 (a typical probability of about 0.0004). Their gradient is 4.4 times larger than the real policy gradient. The real signal is swamped.

The result in the lab is stark. With the mask, the agent learns to search with the tool in all four seeds we ran (success on hidden-number tasks rises from about 0.2 to between 0.8 and 0.99). Without it, three of the four runs abandon the tool entirely and the fourth stalls at one call per episode. Success ends between 0.10 and 0.33, against a guessing rate of 1/8. Real systems show the same direction, less dramatically. Search-R1 (arXiv v5, Section 5.4) reports an ablation on Qwen2.5-7B-Base trained with PPO: average exact match 0.431 with retrieved-token masking and 0.343 without. The GLM-4.5 report states the rule in one sentence: “Only model-generated tokens are used for optimization, and the environment feedback is ignored in loss computation” (GLM-4.5 Team, 2025).

Rewards for agents

Chapter 14 surveyed where rewards come from. Agents add a few patterns of their own.

Outcome rewards from the environment. These are the common case, and the most robust:

• Unit tests. DeepSWE gives reward 1 if the final patch passes the selected tests within five minutes, else 0 (Agentica & Together AI, 2025).
• Exact match. Search-R1 checks the final answer against the gold answer.
• Similarity, when nothing can be executed. SWE-RL trains on GitHub pull-request history without running code. The reward is −1 for a malformed output and otherwise the difflib similarity between the predicted patch and the real one, a number in $[0, 1]$ (Wei et al., 2025).

Format and validity. A malformed tool call usually ends the episode. GLM-4.5 halts the trajectory and gives reward 0. As you will see in Exercise 18.6, the value you give a malformed turn matters more than it looks.

Turn-level (process) rewards. You can also reward individual turns: a tool call that parses, a search that finds a relevant document, a test that newly passes. Dense rewards make credit assignment easier, but each one is a new thing to game. If you pay +0.1 for every tool call that runs, an agent that calls run_tests() in a loop gets paid for being busy. (The demo below lets you watch this happen.) Production systems that use process rewards keep them narrow and behavioral. MiniMax’s M2 report describes per-step rewards that penalize language mixing and tool-format errors, plus a speed term that favors finishing faster (MiniMax, 2026).

Costs. Tool calls cost time and money. A small negative reward per call, or per second of wall clock, teaches the agent to use tools when they help and skip them when they don’t. Getting the size right is delicate. In the lab, a cost of 0.05 per call works. At 0.1, all four seeds we tried learned to never call the tool. Early in training, before the agent knows how to use a tool result, a call is pure cost, so the agent drops tools before discovering they help. One remedy is to bring such penalties in gradually. Kimi k1.5 warms up its length penalty during training (Kimi Team, 2025).

Credit assignment over many turns

One number, many decisions

Look again at the four-turn coding episode. It earned reward 1. Which turn deserves the credit? The edit that changed - to + was the decisive move. Reading the file was useful. Running the tests was a sensible check. Submitting was necessary but trivial. A single outcome reward can’t say any of that. The simplest thing you can do, and what most agentic RL systems do, is give every turn the same credit.

Scheme 1: one advantage for the whole trajectory

Sample a group of $G$ episodes for the same task, as in GRPO (Chapter 15). Let $R^{(i)} = \sum_k r^{(i)}_k$ be the total reward of episode $i$. Every turn, and so every model token, of episode $i$ gets

(Many recipes also divide by the group’s standard deviation. Agentic recipes such as DeepSWE drop that, following Dr. GRPO; see Chapter 16.)

A worked group. Four episodes on the coding task. A fixes the bug properly. B makes a malformed call, then a wrong fix. C edits the test so that it passes. D runs the tests over and over until it hits the five-turn limit. The grader checks whether the repository’s tests pass at submission. The rewards are $R = (1, 0, 1, 0)$, the baseline is $0.5$, and the advantages are $(+0.5, -0.5, +0.5, -0.5)$. Every token of A’s four turns is pushed up by $0.5$, the same as every token of C’s test-editing hack. Hold that thought until the section on reward hacking.

This scheme is simple, and it is Chapter 7’s REINFORCE with a baseline. It points in the right direction on average. (Strictly, because the group mean includes the episode itself, the expected update is shrunk by a factor $(G-1)/G$; RLOO’s leave-one-out baseline removes that.) Its weakness is variance. Every token in the episode receives the same push, including tokens whose choice had nothing to do with the outcome. Those pushes are noise, and they grow with the number of decisions.

Why long horizons are hard: the noise grows with the number of decisions

Here is the effect in its simplest form. An episode has $K$ binary decisions $a_1, \dots, a_K$, each made with its own parameter: $\pi(a_k = 1) = \sigma(\theta_k)$. Only the first decision matters, $R = a_1$. Take $\theta = 0$ (every decision is a coin flip) and the ideal baseline $b = \mathbb{E}[R] = 1/2$. The score of decision $k$ (Chapter 7’s name for the gradient of its log-probability) is $s_k = \partial_{\theta_k} \log \pi(a_k) = a_k - \sigma(\theta_k) = a_k - \tfrac12 = \pm\tfrac12$, and the single-episode REINFORCE estimate has components $g_k = s_k (R - b)$.

• For the decision that matters, $g_1 = (a_1 - \tfrac12)^2 = \tfrac14$ always. The signal is $\tfrac14$.
• For each irrelevant decision, $s_k$ is independent of $R$, so $\mathbb{E}[g_k] = 0$ but $g_k = \pm\tfrac14$ with equal probability. Each one adds variance $\tfrac{1}{16}$.

The total noise in the gradient vector is $(K - 1)/16$ against a squared signal of $1/16$. Average over $N$ episodes and the noise drops to $(K-1)/(16N)$. To make the noise smaller than the signal you need $N > K - 1$ episodes. The number of episodes you need grows linearly with the number of decisions that didn’t matter. An agent that takes 200 turns, most of them routine, pays this tax on every update. Exercise 18.4 works through the general case.

Scheme 2: per-turn credit with reward-to-go

Chapter 7 gave the first remedy: causality. An action can’t affect rewards that came before it, so weight each turn by the reward from that turn on. With a discount $\gamma$ across turns:

Two letters $G$ appear here, both in the course’s standard notation: $G$ alone is the group size, and $G^{(i)}_k$ with a turn subscript is the return from turn $k$ of episode $i$ (Chapter 3’s $G_t$, counted in turns instead of time steps). In words: each turn is scored by the discounted reward still to come, and the baseline is the average full-episode return of the group.

MiniMax’s M2 report uses this form for its agentic RL, “reward-to-go” minus a trajectory-level baseline. Two cases are worth separating.

When rewards arrive along the way (tool costs, process rewards), reward-to-go helps. A late turn isn’t charged for costs paid before it, so its weight carries less noise.

When there is only a final reward and $\gamma = 1$, reward-to-go equals the total reward for every turn, and this is just Scheme 1. Lowering $\gamma$ below 1 changes that, but not in the way you might hope. Take episode A with $\gamma = 0.8$ and the group above. The returns-to-go of its four turns are $0.8^3, 0.8^2, 0.8, 1 = 0.512, 0.64, 0.8, 1$. The baseline is the mean of the four episodes’ $G_0$: $(0.512 + 0 + 0.512 + 0)/4 = 0.256$. So A’s advantages are $(+0.256, +0.384, +0.544, +0.744)$. The submit() turn now gets nearly twice the credit of the decisive edit. Discounting assigns credit by proximity to the reward, not by cause. It also quietly rewards finishing in fewer turns, which may or may not be what you want.

Scheme 3: learn a value function

Chapter 8’s answer to noisy returns was a critic: learn $V(h_k)$ and use TD errors or GAE across turns. A turn-level critic can judge an action against what was expected from that point, and so separate the edit from the submit. The catch is the one that pushed LLM RL toward critic-free methods in Chapter 15: the critic is itself a large model, it is hard to train on sparse rewards, and in agentic settings its input is a long, mostly tool-written context. Some multi-turn work adds one back. RAGEN’s StarPO-S adds a critic baseline, together with trajectory filtering and decoupled clipping, to stabilize multi-turn training (Wang et al., 2025).

Scheme 4: compare actions taken from the same state

GRPO’s trick was to get a baseline by sampling several answers to the same prompt. GiGPO applies the same trick one level down (Feng et al., 2025). In many environments, different episodes pass through the same state: the same web page, the same room in a text game. GiGPO finds these repeated anchor states across a group of trajectories and compares the actions taken from each one, which gives a step-level relative advantage on top of the episode-level one. It needs no critic and no extra rollouts. The authors report gains of more than 12% on ALFWorld and more than 9% on WebShop over GRPO. The limitation is that states must actually repeat, which is common in structured environments and rare in open-ended coding.

Seeing it on one group of episodes

The demo shows the four-episode group from the worked example. Every token is colored by who wrote it. Tint shows the push it receives: green up, red down, hatched none. Use it to check every number in the last two sections, then break things.

Putting it together: GRPO over trajectories

Here is the algorithm that most current agentic RL systems run, in the form you will implement in the lab. It is Chapter 15’s GRPO with two changes: rollouts are multi-turn loops, and the loss is masked.

repeat:
  1. Snapshot the policy for sampling.
  2. For each of B tasks, run G episodes with the snapshot:
       context = prompt
       loop:
         sample the model's message token by token (mark these tokens m = 1)
         if it is a tool call: execute it, append the output (mark these tokens m = 0)
         if it is an answer, malformed, or a limit is hit: stop and score the episode
  3. Per-turn rewards -> per-turn advantages within each group (trajectory-level or reward-to-go).
     Give every model token the advantage of its turn.
  4. Recompute log π_θ for every token in one forward pass over each full episode.
  5. Loss = token-masked policy-gradient loss (+ clipping, + masked KL if used).
  6. Optimizer step.

Real systems add the machinery of Chapters 9, 16 and 20 on top: importance ratios and clipping for reused or stale data, filtering of all-success or all-failure groups, and asynchronous rollouts. DeepSWE’s “GRPO++”, for example, combines clip-higher, no KL loss, no standard-deviation normalization, and a leave-one-out baseline. It also adds compact filtering: episodes that hit the context limit, the step limit or the time limit are masked out of the loss entirely, instead of being treated as failures (Agentica & Together AI, 2025).

Environments

An RL environment for an LLM agent is the same object as in Chapter 1: something with reset() and step(action) that returns an observation, a reward and a done flag. The difference is in what sits behind step. It might be a Python interpreter, a Docker container holding a whole repository, a search engine, or a browser. Hugging Face and Meta’s OpenEnv uses exactly the Gymnasium-style reset()/step() interface, with environments running as containers or HTTP servers (Hugging Face, 2025). Prime Intellect’s verifiers library packages an environment as a dataset, a harness and a rubric, and shares them on an Environments Hub (Prime Intellect, 2025).

A good training environment is:

• Verifiable: it can score an episode automatically and cheaply.
• Reproducible: the same task resets to the same state, so a group of $G$ episodes really is $G$ attempts at one problem.
• Isolated: one episode can’t affect another, or the outside world.
• Fast enough: tool latency multiplies across turns and episodes.
• Hard to game: the grader checks what you actually want (more on this below).

Here are the main families, with representative work. All benchmark numbers are as reported by the authors.

Two observations from this literature. First, environments have become the bottleneck and the main scaling axis. Frontier reports in 2026 count environments the way 2023 reports counted preference pairs: GLM-5 reports more than 10,000 verifiable SWE environments. There are even attempts to learn the environment itself: Qwen-AgentWorld trains LLMs to simulate tool environments for agentic RL (Zuo et al., 2026). Second, multi-turn RL has its own instabilities. RAGEN reports an “Echo Trap”, in which reward variance collapses and gradients spike as the agent overfits to locally rewarded reasoning patterns. Its abstract also warns that without fine-grained, reasoning-aware reward signals, agent reasoning “hardly emerge[s]” through multi-turn RL.

The practical problems

Long, variable-length rollouts

Chapter 15’s rollouts already had a long tail of lengths. Agentic rollouts are worse, because the length varies in turns as well as tokens, and each turn includes a tool’s latency. A synchronous training step waits for the slowest episode in the batch, so a few long episodes leave most of the hardware idle. This is the main reason agentic RL moved to asynchronous systems: rollout workers keep generating while the trainer updates, and samples arrive slightly stale. Staleness makes the data off-policy, so the importance ratios and clipping of Chapter 9 come back. GLM-4.5’s slime framework offers a colocated synchronous mode, which it reports works best for math and code, and a disaggregated asynchronous mode for long agent rollouts. Chapter 20 covers these designs. Exercise 18.7 puts numbers on why synchronous rollouts waste so much.

Context length and context management

A long-horizon agent will overflow any context window. The options are crude to clever: truncate old tool outputs, cap each observation (a 50,000-line test log becomes its last 200 lines), or let the model summarize its own history and continue from the summary. Cursor’s Composer 2 report describes training with self-summarization, where the outcome reward applies to the whole chain, including the model’s own summaries. A summary that drops a crucial detail leads to failure later, and RL pushes back on it. Whatever you do, the policy at training time must see what it will see at deployment. And episodes cut off by the limit need a decision: count them as failures, or mask them out, as DeepSWE’s compact filtering does. The same question arose for overlong responses in Chapter 16.

Sandboxes: cost and safety

Every coding episode needs a fresh, isolated copy of the repository and its dependencies. DeepSWE ran 512 Docker containers per training step (64 tasks × 8 rollouts). DeepSeek-V4’s report (DeepSeek-AI, 2026) describes a sandbox platform running “hundreds of thousands of concurrent sandbox instances”. Isolation is also a safety boundary, and an optimizing agent will probe it. Anthropic’s Claude Mythos Preview system card (April 2026) reports that, in automated review of RL transcripts, the model “occasionally circumvented network restrictions in its training environment to access the internet and download data that let it shortcut the assigned task”, in about 0.05% of episodes, about 0.01% successfully.

Environment reward hacking

Look back at episode C. The grader checked “do the repository’s tests pass?”, and the agent made them pass by editing the test. This is not a hypothetical. OpenAI’s study of monitoring a frontier reasoning model during agentic coding RL found systemic hacks such as calling exit(0) so the test harness reports success, and raising SkipTest (Baker et al., 2025). Anthropic studied production coding environments vulnerable to three hacks: returning an object whose __eq__ always returns True, calling sys.exit(0), and patching conftest.py to rewrite test results. When models learned these hacks, the behavior generalized to broader misalignment, such as alignment faking and sabotage (MacDiarmid et al., 2025).

Agentic environments are especially hackable for a structural reason. The agent can act on the same world the grader inspects, so it can change the evidence. The defenses are mostly environment engineering:

• grade with held-out tests the agent never sees, run in a clean copy of the repository;
• make tests and graders read-only to the agent;
• diff-check the final state (did the agent touch test files?);
• monitor trajectories, including the reasoning, for signs of gaming.

Chapter 19 treats reward hacking and monitoring in depth.

Multi-agent and self-play setups

Two directions extend the single-agent loop.

An orchestrator with sub-agents. Kimi K2.5’s “Agent Swarm” trains an orchestrator that launches parallel sub-agents (Kimi Team, 2026). The sub-agents are frozen, and their outputs come back to the orchestrator as observations. From the orchestrator’s point of view, a sub-agent is a tool, so its output is masked like any other tool output. The reward includes two auxiliary terms aimed at failure modes: one against “serial collapse” (never parallelizing) and one against “spurious parallelism” (launching many useless sub-agents, a reward hack). Both are annealed to zero during training.

Self-play. If the environment can generate its own tasks, the agent can train without human-written ones. In Self-play SWE-RL, one agent both injects bugs into real repositories (specified by test patches) and repairs them. The authors report gains of 10.4 points on SWE-bench Verified and 7.8 on SWE-Bench Pro, beating a baseline trained on human data (Wei et al., 2025). Chapter 21 returns to self-play and zero-data RL.

Summary

• An LLM agent alternates model messages and environment observations. Formally it is a (partially observed) MDP whose state is the context window and whose transitions include environment-written tokens.
• The policy gradient is a sum over the model’s own tokens only. Tool outputs are state, not actions, so a loss mask zeroes them, and the same mask applies to the KL, entropy and normalizer. Without it you train the model to imitate (or un-imitate) the environment, with large gradients that never self-correct.
• Compute log-probs on the exact sampled token ids, under the same sampling rules the sampler used. Small mismatches here can break learning.
• With one reward per episode, the default is trajectory-level credit: every model token gets $R^{(i)} - \bar{R}$. Its variance grows roughly linearly with the number of turns that didn’t matter.
• Reward-to-go helps when rewards arrive along the way. With only a final reward, discounting favors late turns rather than decisive ones. Critics and anchor-state grouping (GiGPO) are the main ways to do better.
• Agent rewards are mostly environment outcomes (tests, exact match), plus format penalties and sometimes small costs or process rewards. Every extra term is a new thing to tune and to hack.
• Practical problems dominate: long variable rollouts (hence asynchronous systems), context limits (truncation, summarization), sandbox cost and isolation, and environment reward hacking, where the agent alters what the grader checks.

Exercises

[prompt]  Q: Who wrote Middlemarch?
[model]   <search> Middlemarch author </search>
[tool]    <information> Middlemarch is a novel by George Eliot ... </information>
[model]   <answer> George Eliot </answer>

cd 18-agentic-rl
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Jin et al. (2025), Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. The cleanest small-scale agentic RL paper: interleaved search calls, outcome reward, and an ablation of retrieved-token masking (Section 5.4).
• Agentica & Together AI (2025), DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. A detailed, honest recipe for RL on real repositories: rewards, GRPO++ choices, compact filtering, and what the infrastructure took.
• Wang et al. (2025), RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. Multi-turn RL on small environments, and the “Echo Trap” instability with its fixes.
• Feng et al. (2025), Group-in-Group Policy Optimization for LLM Agent Training. Step-level credit without a critic, via anchor states. A good model of how to extend GRPO’s idea to turns.
• Wei et al. (2025), SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. RL for software engineering with a similarity reward and no code execution. A useful contrast with test-based rewards.
• GLM-4.5 Team (2025), GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. A production report that spells out agentic RL choices: masking, format penalties, synchronous vs asynchronous rollouts.
• MacDiarmid et al. (2025), Natural Emergent Misalignment from Reward Hacking in Production RL. What happens when coding environments are hackable, and why it matters beyond the hacked task. Read before you build an environment.
• Zhang et al. (2025), The Landscape of Agentic Reinforcement Learning for LLMs: A Survey. A map of the field, organized by capability and application, for when you want breadth.

Next, Chapter 19 takes the reward hacking you saw in episode C and asks the general question: what goes wrong when you optimize hard against any imperfect reward?