RL systems for LLMs

In Chapter 15 you wrote GRPO in about forty lines of PyTorch, and it trained a tiny model on your laptop in three minutes. Now someone hands you 64 GPUs and a 7-billion-parameter model and asks for the same thing. You launch it and watch the GPU monitor. Most of the time, most of the GPUs are doing nothing. They are waiting for one response, out of hundreds, that is still writing its 30,000th token.

You fix that by letting generation run ahead of training. The run gets three times faster. Then, a few hundred steps in, the reward collapses for no visible reason. The cause, it turns out, is that the program that generated the tokens and the program that trained on them computed slightly different probabilities for the same tokens with the same weights, and your “on-policy” algorithm had been quietly off-policy all along.

Both surprises are what this chapter is about. At scale, RL for LLMs is as much a systems problem as an algorithms problem, and the systems choices leak into the math: they decide how stale your data is, which importance ratios you need, and which samples you silently throw away. You will see where the time and memory go, why generation dominates, how synchronous and asynchronous designs trade speed for staleness, and how the community found and fixed the train–inference mismatch in 2025. Along the way, ideas from Part II come back: off-policy learning (Chapter 5), importance sampling and trust regions (Chapter 9), and control variates (Chapter 7).

One RL step, seen by the hardware

Here is one step of GRPO or PPO as the hardware sees it. Nothing in it is new. What is new is that each line runs on different software, often on different machines.

repeat:
  1. GENERATE  sample G responses for each of B prompts with the current policy     (inference engine)
  2. SCORE     reward each response: verifier, unit tests in a sandbox, or a reward model   (reward workers)
  3. PREPARE   log-probs of every sampled token under the old policy and the reference;
               values from the critic (PPO only); advantages                         (trainer, forward passes)
  4. TRAIN     a few gradient steps on the clipped objective                          (trainer)
  5. SYNC      copy the new weights into the inference engine                         (network or memory copy)

Each stage has its own specialized software:

Why not use one program for everything? Because generating text and training on text are different workloads, and the code that is fastest for one is slow for the other. The next section shows why.

Why generation dominates

Training is parallel, generation is not

When the trainer computes $\log \pi_\theta(y_t \mid x, y_{\lt t})$ for a 2,000-token response, it already knows all 2,000 tokens. One forward pass handles them all at once (this is teacher forcing). The GPU multiplies large matrices, which is what it is built for.

Generation can’t do that. Token 1,001 depends on token 1,000, which the model has not written yet. So the engine runs 2,000 forward passes in a row, each producing a single token per sequence. Each pass must read every weight of the model from GPU memory, and then does only a little arithmetic with each one.

A worked example: the memory-bandwidth wall

Take a 7-billion-parameter model in bf16 (2 bytes per parameter), so 14 GB of weights, on an NVIDIA H100 SXM. The H100 can read memory at about 3.35 TB/s and do about 989 trillion bf16 operations per second (dense).

One decode step at batch size 1. Reading 14 GB at 3.35 TB/s takes

So one sequence gets at most about 240 tokens per second, no matter how fast the arithmetic units are. And the arithmetic needed is small: a forward pass costs about 2 operations per parameter per token, $2 \times 7\times10^9 = 1.4\times 10^{10}$ operations, which the H100 could do in 0.014 ms. The arithmetic units sit idle about 99.7% of the time. Decode at batch size 1 is memory-bandwidth-bound.

The fix is batching. If 64 sequences decode together, the weights are read once per step and used 64 times. The step still takes about 4.2 ms, but now produces 64 tokens. The step only becomes limited by arithmetic when the batch is large enough that the math takes as long as the memory read:

So you want hundreds of sequences in flight at once. What stops you is memory, and specifically the KV cache.

The KV cache

During decode, attention at step $t$ needs the keys and values of all earlier tokens. Recomputing them every step would be wasteful, so the engine stores them: the KV cache. Its size per token is

where the 2 counts one key and one value. For Qwen2.5-7B (28 layers, 4 KV heads of dimension 128, bf16) that is $2 \times 28 \times 4 \times 128 \times 2 = 57{,}344$ bytes, 56 KiB per token. That sounds small until you multiply:

The weights of that model take about 15 GB (Qwen2.5-7B actually has 7.6 billion parameters, not exactly 7). Weights plus 60 GB of KV cache almost fill an 80 GB H100. So at this response length, memory caps the batch at about 64, well short of the 300 you wanted. And the KV cache also has to be read every decode step. Near the end of those 16k-token responses, each step reads 15 GB of weights plus 60 GB of cache, about 22 ms, giving roughly $64 / 0.022 \approx 2{,}800$ tokens per second. At 1,000 tokens of context the same GPU manages about 11,000. Long chains of thought make every token slower, not just more numerous.

The long tail

Response lengths in reasoning RL are wildly uneven. Most responses to a math problem might be 2,000 tokens, while a few run to the 32,000-token limit. In a synchronous step, the step ends when the last response ends. As the batch drains, fewer and fewer sequences are left in flight, and since a decode step costs about the same whether it serves 64 sequences or 3, the GPU spends the tail of the step doing a few percent of its possible work.

Here is how bad it gets. Suppose lengths are exponentially distributed with mean $m$ and you run $N$ sequences in parallel, one per slot, then wait for all of them. The expected longest is $m \cdot H_N$, where $H_N = 1 + \tfrac12 + \dots + \tfrac1N \approx \ln N + 0.58$ (Exercise 20.4 derives this). The slots are busy for an average of $m$ each out of $m H_N$, so the utilization is

For $N = 64$, $H_{64} \approx 4.74$: the generation hardware does useful work about 21% of the time. And real length distributions have heavier tails than the exponential.

Put the pieces together: generating a token is slower than training on it, long contexts slow it further, and the tail wastes most of the rest. Systems papers on rollout efficiency report generation taking the bulk of each RL step; for example, the APRIL paper (Zhou et al. 2025) states that rollout generation accounts for more than 90% of total runtime in their setting. The exact share depends on model, lengths and hardware, but the conclusion doesn’t: in RL for LLMs, you optimize the generation stage first.

Memory accounting: where the bytes go

Before deciding how to place things on GPUs, you need to know how big they are. The rule of thumb comes from the ZeRO paper (Rajbhandari et al. 2019).

16 bytes per trained parameter

Training with mixed-precision Adam keeps five numbers per parameter:

A frozen model that only runs forward passes needs just its weights: 2 bytes per parameter in bf16. These numbers exclude activations (the intermediate values saved for the backward pass, which grow with batch size and sequence length) and the KV cache. Real runs need noticeably more than this table; treat it as a floor.

PPO vs GRPO for a 7B policy

Now count the models. PPO-based RLHF (Chapter 12) has four: the policy (trained), the critic (trained, usually the same size), the reference (frozen) and the reward model (frozen). GRPO with a verifiable reward (Chapter 15) has no critic and no reward model; many 2025 recipes also drop the KL term, and with it the reference model.

Three things to notice. First, the critic is the single biggest saving when you go from PPO to GRPO: it costs as much as the policy because it is trained too. This is the memory half of Chapter 15’s “no critic” argument. Second, none of the full fine-tuning rows fits on one 80 GB GPU even before activations and KV cache. Third, with LoRA the reference model costs nothing: the reference is the base model, and the base model is already in memory. Switch the adapter off and you have $\pi_\text{ref}$.

How do the 252 GB fit at all? By sharding: ZeRO stage 3 or PyTorch’s FSDP splits the weights, gradients and optimizer states of each model evenly across $n$ GPUs and gathers each layer’s weights just in time for its computation. With 8 GPUs, the PPO setup above needs about $252 / 8 \approx 32$ GB of states per GPU, leaving the rest for activations. Very large models add tensor parallelism (splitting individual matrices across GPUs) and pipeline parallelism (splitting layers), which is what Megatron-LM provides.

Placement: colocated or disaggregated

You have a fixed set of GPUs, and two very different jobs. There are two basic ways to share the GPUs between them.

Colocated (also called a hybrid engine; DeepSpeed-Chat and verl popularized it): every GPU does both jobs, in turns. During generation the GPUs run the inference engine; then the engine releases its KV-cache memory, the same GPUs switch to training, and afterwards the new weights are resharded back into the engine’s layout. verl’s paper (Sheng et al. 2024, “HybridFlow”) calls its version the 3D-HybridEngine.

Disaggregated: some GPUs only generate, others only train. After each training step, new weights travel over the network to the generation GPUs.

How expensive is moving the weights? For a 7B model in bf16, 14 GB. Over a 400 Gb/s (50 GB/s) network link that is about

tiny next to a generation phase that takes minutes. For a 1-trillion-parameter model (2 TB in bf16) one link would need 40 seconds, so frontier systems shard the transfer across many links at once. Weight sync is usually not the bottleneck, but it is not free.

Production systems support both. The GLM-4.5 report, for instance, describes its slime framework running colocated and synchronous for math and code RL, and disaggregated and asynchronous for long agent rollouts. The choice comes down to the question of the next section: are you willing to train on slightly stale data?

Synchronous vs asynchronous RL

The synchronous bubble

In synchronous RL every sample is generated by the current policy, then the policy is updated, then the next batch is generated. It is exactly on-policy, the setting in which REINFORCE, PPO and GRPO were derived. It is also wasteful in two ways. Generation hardware waits for the slowest response (the long tail), and in a disaggregated setup, generation and training hardware also wait for each other.

Let generation run ahead

The fix is to let the generators keep going while the trainer works. A response started while the policy was at version $v$ (after $v$ training steps) is tagged $v$. If the trainer uses it in the step that updates version $v'$ to $v'+1$, its lag (or staleness) is $v' - v$. Lag 0 is on-policy. Lag 2 means the sample came from the policy of two updates ago.

To keep lags under control, systems bound how far generation may run ahead. AReaL (Fu et al. 2025) uses a simple admission rule: with batch size $B$ and a maximum staleness $k$, a new generation may start only while

where $v$ is the trainer’s current version. With $k = 0$ you may start one batch, then must wait for the update: synchronous RL. With $k = 1$ the generators may also start the next batch with the current weights while the trainer works on this one: the popular one-step off-policy design.

Here is a hand-sized example (it is also a test in the lab). Two generation lanes, batches of $B = 2$, two training steps, each taking 1 time unit. Responses are numbered in the order they start, r0, r1, r2, …, and take 1, 3, 1, 1, 1, … time units.

• Synchronous ($k = 0$). r0 (length 1) and r1 (length 3) start at $t = 0$. The batch is complete at $t = 3$, and the trainer runs step s0 from 3 to 4. Only then may r2 and r3 start; they run from 4 to 5, and step s1 runs from 5 to 6. Total: 6 units. The lanes worked $1 + 3 + 1 + 1 = 6$ of their $2 \times 6 = 12$ lane-units: 50% utilization.
• Asynchronous ($k = 1$, so up to $(0 + 1 + 1) \times 2 = 4$ responses may start at version 0). Lane A finishes r0 at $t = 1$ and immediately starts r2 (tagged version 0). r2 finishes at $t = 2$, so the trainer has two responses (r0, r2) and runs s0 from 2 to 3. Meanwhile lane A starts r3, the fourth and last response allowed at version 0. At $t = 3$, r1 and r3 finish and the trainer, now at version 1, runs s1 on them from 3 to 4. Also at $t = 3$, the admission limit has risen to $(1 + 1 + 1) \times 2 = 6$, so both lanes start r4 and r5 for the next step. The lanes are never idle. Total: 4 units, 100% lane utilization, and step s1 trained on two samples with lag 1.

Drawn as a timeline (each character is a quarter of a time unit, dots are idle time):

Synchronous, k = 0
time     0   1   2   3   4   5   6
lane A   [r0]............[r2]....
lane B   [----r1----]....[r3]....
trainer  ............[s0]....[s1]

Asynchronous, k = 1
time     0   1   2   3   4
lane A   [r0][r2][r3][r4]
lane B   [----r1----][r5]
trainer  ........[s0][s1]

The demo below runs the same simulation at a larger scale.

The lower panel shows the real trade-off: every step of $k$ buys throughput and costs lag, and past the point where the slowest component is saturated, extra $k$ buys nothing. Two more things the demo shows that are easy to miss:

• The admission rule bounds how far generation runs ahead, not how old every sample is. A very long response started early can finish many versions later, so the maximum lag can exceed $k$ (with the defaults and $k = 1$, the demo’s max lag is 5). Systems handle these stragglers by dropping them (GLM-5 drops samples whose generating weights lag by more than a threshold), by interrupting them, or by accepting them with importance weights.
• Dropping stale samples drops the long ones. Short responses finish fast and are trained on quickly; the samples that age past $k$ are the long ones. If long responses are disproportionately the hard, careful, correct ones (or the rambling, wrong ones), dropping them biases training toward one kind of behaviour. The first-in, first-out queue has the same flavour of bias: early batches over-represent short responses. Exercise 20.9 explores this.

Correcting for stale samples

A sample with lag $d$ was drawn from $\pi_{\theta_{v-d}}$, but the gradient we want is for the current $\pi_\theta$. This is exactly the off-policy problem of Chapter 5: a behaviour policy $\mu$ collected the data and we want to improve a different target policy. And Chapter 9 already gave the tool, importance sampling:

PPO’s ratio $\rho_t(\theta) = \pi_\theta(y_t \mid \cdot) / \pi_\text{old}(y_t \mid \cdot)$ is this correction applied per token, with $\pi_\text{old}$ playing the role of $\mu$. So staleness is handled if $\pi_\text{old}$ really is the policy that sampled the token. That is the practical rule: record the log-probability of every token at the moment it is sampled, and use that as the behaviour log-prob. If the trainer instead recomputes “old” log-probs with its current weights, the ratio corrects for the wrong distribution.

There is a subtlety. PPO’s clip keeps $\pi_\theta$ close to $\pi_\text{old}$, which is a trust region. When $\pi_\text{old}$ is several versions behind, the clip keeps pulling the new policy back toward an outdated one. AReaL’s decoupled PPO objective separates the two roles: the behaviour policy $\pi_\text{behav}$ (which sampled the data) gets an importance weight, and a recent proximal policy $\pi_\text{prox}$ (in practice, the weights just before the current update) anchors the clip:

How much staleness is tolerable? Noukhovitch et al. (2024, “Asynchronous RLHF”) studied exactly this. They found online DPO the most robust of the algorithms they tried to off-policy data, with robustness growing with model size, and trained an instruction-following chatbot from Llama 3.1 8B about 40% faster than a synchronous run with matching final performance. AReaL reports up to 2.77× end-to-end speedups over synchronous systems on math and code reasoning. In practice the bounds are small: AReaL uses a maximum staleness of 4 for coding and 8 for math, and ScaleRL (below) uses a maximum off-policyness of 8.

Two more tricks for the long tail

Partial rollouts. Kimi k1.5 (Kimi Team 2025) caps the work per iteration: a response that has not finished by the end of an iteration is paused and resumed in a later one. A single response then contains segments generated by different policy versions.

In-flight weight updates. PipelineRL (Piché et al. 2025) goes further: the inference engine receives new weights in the middle of generating a sequence and continues with them, keeping its KV cache. The authors report about 2× faster learning on 128 H100s while keeping the data close to on-policy. Olmo 3 and Magistral describe the same mechanism in their RL infrastructure. As with partial rollouts, one response is now a patchwork of policy versions. The fix is the same practical rule as before, applied per token: each token’s behaviour log-prob is whatever the engine used when it sampled that token.

The train–inference mismatch

Same weights, different probabilities

Now the second surprise from the opening. Take a fully synchronous run: the trainer updates the weights, copies them exactly into the inference engine, the engine samples a batch, and the trainer trains on it. Lag zero everywhere. Surely this is on-policy?

It isn’t. The inference engine and the trainer are different programs. They compute the same mathematical function with different kernels (fused attention, different matrix-multiplication tilings), different reduction orders, sometimes different precision. Floating-point addition is not associative, so $(a + b) + c$ and $a + (b + c)$ can differ in the last bits. Run those differences through dozens of layers and a softmax, and the engine’s probability for a token, call it $\mu_\theta(y_t \mid \cdot)$, differs from the trainer’s $\pi_\theta(y_t \mid \cdot)$, with identical weights.

The sources of mismatch, roughly from smallest to largest:

• Batch-dependent arithmetic. An inference server’s kernels may compute a sequence’s logits differently depending on what else is in the batch. Thinking Machines (He et al. 2025, “Defeating Nondeterminism in LLM Inference”) sampled 1,000 completions from Qwen3-235B at temperature 0 and got 80 distinct ones; with batch-invariant kernels, all 1,000 were identical.
• Low precision. bf16 keeps only 8 significant bits. Numbers between 16 and 32 are spaced $16 \times 2^{-7} = 0.125$ apart, so a logit near 20 can be off by up to 0.0625 from rounding alone. Shifting one logit by 0.0625 multiplies that token’s (small) probability by about $e^{0.0625} \approx 1.065$: a 6.5% change. fp16 has 11 significant bits: spacing $0.0156$, worst case $e^{0.0078} \approx 1.008$. fp32: about $2\times10^{-6}$. MiniMax-M1’s report found the LM head (the last layer, which produces the logits) was the culprit in their model, and computing it in fp32 raised the correlation between training-mode and inference-mode probabilities from about 0.9 to about 0.99.
• MoE routing flips. In a mixture-of-experts model, a router picks the top-$k$ experts for each token. A tiny numerical difference can change which experts are chosen, and then the two programs compute genuinely different functions.
• Deliberate differences. Rollouts in FP8 or FP4 to save memory and time, or top-$p$ sampling that truncates the distribution.

And small per-token differences compound over long responses. If each token’s log-ratio $\log(\pi/\mu)$ has a random error of about 0.01, a 10,000-token response has a sequence log-ratio with standard deviation about $0.01 \times \sqrt{10{,}000} = 1$. The trainer and the engine then disagree about that whole response’s probability by a factor of around $e^{\pm 1}$, nearly 3 either way.

Why it biases the gradient

The engine samples $y \sim \mu_\theta$, but the trainer computes the REINFORCE-style update as if $y \sim \pi_\theta$:

The gap is a bias, and bias does not shrink with batch size. Sampling noise averages away as you add samples; a systematic error stays. Small biases accumulate step after step, and the policy can drift into regions where $\pi$ and $\mu$ disagree even more, which makes the bias bigger. Thinking Machines’ RL experiment shows the pattern: without a correction, the reward collapsed partway through training at the same time as the measured KL between sampler and trainer spiked; with an importance-weight correction, and with bitwise-identical sampler and trainer, training was smooth.

The fix: (truncated) importance sampling

This is another off-policy correction, and the tool is again the importance weight, now between trainer and sampler. Qwen’s analysis (Zheng et al. 2025, “Stabilizing Reinforcement Learning with LLMs”) makes the bookkeeping clean. The ratio a token really needs, the current trainer policy over what actually sampled it, factors into two pieces:

PPO and GRPO already handle the second factor with $\rho_t(\theta)$ and its clip. The first factor is the missing piece. Call it $w_t$. It can be computed: the engine reports $\log \mu_{\theta_\text{old}}(y_t \mid \cdot)$ as it samples, and the trainer computes $\log \pi_{\theta_\text{old}}(y_t \mid \cdot)$ in its “prepare” pass.

Multiplying by the importance weight removes the bias (exactly with the whole-response weight, approximately with per-token weights $w_t$; the derivation below says how approximately). But importance weights have a nasty property: rare tokens can get huge weights (the sampler barely ever picks a token that the trainer likes), and huge weights mean huge variance. Truncated importance sampling (TIS), proposed for this problem by Yao et al. (2025), caps them: use $\min(w_t, C)$ with a constant $C$ such as 2. Olmo 3’s RL objective is a clean example of the late-2025 recipe; in our notation (they write $\pi_\text{vllm}$ for $\mu$ and $\rho$ for $C$):

Everything except $\min(w_t, C)$ is the GRPO-family objective from Chapter 16 (token-level averaging, asymmetric clip, no std normalization of $\hat A$). The weight is a constant: it multiplies the gradient but no gradient flows through it.

What does truncation cost? A little bias, in exchange for bounded variance:

Variants followed quickly. Masked importance sampling zeroes out (rather than caps) the tokens or sequences whose weight is out of range (Liu, Li et al. 2025); IcePop (Ant Group’s Ring-1T, also used by GLM-5) masks tokens whose train/inference ratio leaves a band; Kimi K2.5 and GLM-5 zero the gradient of tokens whose ratio is out of bounds, regardless of the advantage’s sign (PPO’s clip only does so for one sign). By 2026 the PPO clip was, in many production recipes, being replaced by “mask the tokens whose sampler, trainer or staleness ratio is out of band” (Chapter 16 covers the objective side of this).

Or remove the mismatch at the source

Importance weights treat the symptom. Other teams went after the cause:

• More precision where it matters. MiniMax-M1’s fp32 LM head (above). ScaleRL found that computing the logits in fp32 raised the fitted asymptotic pass rate of its 8B runs from 0.52 to 0.61. Qi et al. (2025, “Defeating the Training-Inference Mismatch via FP16”) argue that bf16 rounding is the root cause, and that switching training and inference uniformly to fp16 removes most of the mismatch, giving more stable training across algorithms and frameworks.
• Batch-invariant, matched kernels, so the engine and trainer produce bitwise-identical probabilities (Thinking Machines’ “true on-policy” experiment).
• Routing replay for MoE models. Record which experts the inference engine chose for each token and force the trainer to use the same ones: DeepSeek’s “Keep Routing” (used since DeepSeek-V3-0324, per the V3.2 report) and Rollout Routing Replay (R3; Ma et al. 2025).
• Train in the format you deploy. In 2026 several labs matched rollout and training numerics by construction, for example by running rollouts with the same low-precision expert weights (FP4) used in deployment, or quantization-aware training shared by rollout and trainer.

LoRA for RL

LoRA (low-rank adaptation) freezes the base model and trains a small low-rank update $\Delta W = BA$ for each weight matrix. For supervised fine-tuning it is a well-known way to save memory, at some cost in quality. For RL the trade-off looks unusually good.

Schulman and Thinking Machines (2025, “LoRA Without Regret”) report that for policy-gradient RL, “LoRA fully matches the learning performance of FullFT … even with ranks as low as 1”. Their explanation is an information count. A policy-gradient update is driven by the advantage, which carries about one number, $O(1)$ bits, per episode, however long the episode is; supervised learning gets a target for every token. By their estimate RL absorbs about 1,000 times less information per token, so a tiny adapter has enough capacity to store what RL teaches. They also found the best LoRA learning rate is about 10× the full fine-tuning one, and that LoRA should be applied to all layers, especially the MLP (or MoE) layers. A 2026 paper pushed this to an extreme: TinyLoRA (Morris et al.) trained an 8B Qwen2.5 model to 91% on GSM8K with 13 trained parameters, and found that this works with RL but not with supervised fine-tuning, which needed far larger updates.

The systems payoff is in the memory table above: the policy’s training state shrinks from 112 GB to about 15 GB, the reference model becomes free (adapter off), and weight sync shrinks from the whole model to the adapter. Thinking Machines’ Tinker service builds on this: it exposes a LoRA training API (forward-backward, optimizer step, sampling) on hosted open-weight models, so you write the RL loop and they run the systems.

Scaling RL compute

How much compute should RL get, and does it pay off predictably? The most systematic answer so far is ScaleRL (Khatri et al. 2025, “The Art of Scaling Reinforcement Learning Compute for LLMs”). Over more than 400,000 GPU-hours of ablations, they fit the pass rate $R_C$ on held-out problems after $C$ units of RL compute with a sigmoid,

where $A$ is the asymptote (the best this recipe will ever do), $B$ the efficiency (how fast it gets there) and $C_\text{mid}$ the compute at which half the gain is reached. Two findings matter here. Recipes differ in their asymptote, so a recipe that looks better early can be worse late. And many popular knobs (loss aggregation, advantage normalization, curriculum, the off-policy algorithm) mostly change the efficiency $B$, not the ceiling $A$. Stable recipes let them fit the curve on small runs and extrapolate to a 100,000 GPU-hour run.

Their final recipe is a good snapshot of what this chapter has covered: PipelineRL-style asynchrony with at most 8 steps of off-policyness, fp32 logits, a truncated-importance-sampling loss (CISPO), and no KL term. Both the systems choices and the algorithm choices are in it, because at this scale they are not separable. Chapter 21 follows the scaling story into 2026.

The open-source stacks

You will rarely build all of this yourself. The main open-source frameworks, as of October 2026 (they change fast; check each project’s documentation):

A reasonable path: learn with TRL on one GPU (or with this course’s labs on a laptop), read OpenRLHF or verl to see how a full system fits together, and reach for the asynchronous frameworks when your rollouts get long or multi-turn (Chapter 18).

Summary

• One RL step is generate → score → prepare → train → sync, and each stage runs on specialized software: an inference engine (vLLM, SGLang), a trainer (FSDP, DeepSpeed, Megatron), reward workers and a controller.
• Generation dominates the time. Decode is memory-bandwidth-bound (each token reads all the weights), so it needs large batches; the KV cache (56 KiB per token for a 7B-class model) limits the batch and slows long contexts; and the long tail of response lengths leaves synchronous hardware idle most of the time (utilization about $1/H_N$ for exponential lengths).
• Training state costs 16 bytes per parameter with mixed-precision Adam, frozen models 2. A 7B PPO-RLHF setup needs about 252 GB before activations; GRPO without a critic halves that; LoRA brings it to about 15 GB and makes the reference model free.
• Colocated placement time-shares GPUs between generation and training; disaggregated placement gives each its own pool, which makes asynchrony natural.
• Asynchronous RL lets generation run up to $k$ batches ahead, trading throughput for staleness. It is off-policy RL, corrected by importance ratios computed from log-probs recorded at sampling time. Watch for stragglers and for length bias when dropping stale samples.
• The train–inference mismatch: with identical weights, the sampler’s probabilities $\mu$ differ from the trainer’s $\pi$, which biases the gradient. Fix it with truncated importance sampling $\min(\pi/\mu, C)$ (bounded variance, small bias), or at the source with higher precision, matched kernels, routing replay and matched quantization. Always log the sampler–trainer KL.
• With RL’s low information per episode, LoRA matches full fine-tuning in reported experiments, and RL compute scaling follows sigmoid curves whose asymptote depends on the recipe (ScaleRL).

Exercises

cd 20-rl-systems
python -m pytest -q          # your code: tests fail until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # reference solutions: all pass

Further reading

• Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention. The vLLM paper. Its background sections explain why decode is memory-bound and why KV-cache fragmentation limits the batch size.
• Sheng et al. (2024), HybridFlow: A Flexible and Efficient RLHF Framework. The verl paper: the dataflow view of an RL step, colocated placement and the 3D-HybridEngine.
• Fu et al. (2025), AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. The clearest description of asynchronous RL for reasoning: staleness control and the decoupled PPO objective.
• Noukhovitch et al. (2024), Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. How much off-policy-ness LLM RL can tolerate, algorithm by algorithm.
• Yao et al. (2025), Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. The blog post that named the train–inference mismatch and proposed truncated importance sampling.
• He and Thinking Machines (2025), Defeating Nondeterminism in LLM Inference. Why inference is nondeterministic (batch invariance), with an RL experiment showing what bitwise on-policy training buys.
• Schulman and Thinking Machines (2025), LoRA Without Regret. LoRA for SFT and RL, the information-per-episode argument, and practical learning-rate advice.
• Khatri et al. (2025), The Art of Scaling Reinforcement Learning Compute for LLMs. The ScaleRL sigmoid fits and a recipe that combines the systems and algorithm choices of this chapter.

Next, Chapter 21 steps back to survey what the latest models actually do in post-training, and the open problems worth watching.