# Lab 19: Failure modes you can watch happen

Three small experiments, each reproducing one failure from the chapter:

1. **Overoptimization** (numpy, seconds). Best-of-n against a proxy reward, scored with a gold reward. You compute the curves *exactly* and fit Gao et al.'s functional form.
2. **Reward hacking** (TinyLM, about 2 minutes). A 100k-parameter language model is trained with RL against a plausible-looking reward for the "37+48=" task. It finds the loophole. Then a small KL penalty stops it.
3. **Entropy collapse** (TinyLM, about 2 minutes). Watch the policy's entropy drain away during GRPO, compare the symmetric PPO clip, DAPO's clip-higher and no upper clip at all, and see *which* tokens the upper clip switches off.

Chapter: <https://learn.xhelio.ai/rl/learn/19-failure-modes/>

## Files

| File | What it is |
|---|---|
| `overopt.py` | **Your file, Part 1.** Tasks 1a–1c. |
| `failures.py` | **Your file, Parts 2–3.** Tasks 2–4. |
| `common.py` | Given: the toy reward pool, the two TinyLM rewards (**read `proxy_reward` before running Part 2**), group advantages, and the RL loop `rl_train`. |
| `tinylm.py` | Given: the tiny transformer, the addition task, SFT and sampling (same as the other Part III labs). |
| `test_overopt.py`, `test_failures.py` | Tests. |
| `run.py` | The three experiments; saves `overopt.png`, `hack.png`, `entropy.png`. |
| `solutions/` | Reference solutions. Try not to peek. |

## Tasks

Run one task's tests at a time with `-k`, e.g. `python -m pytest -q -k task1b`.

**Part 1, `overopt.py`**

- **Task 1a `bon_kl(n)`**: $\log n - (n-1)/n$, the KL of best-of-n from the base policy.
- **Task 1b `best_of_n_stats(proxy, gold, n)`**: the expected proxy and gold score of best-of-n from a pool of responses, computed *exactly* by rank weights $w_k = (k/M)^n - ((k-1)/M)^n$. One test checks a 3-response example you can do by hand; another checks against brute-force simulation.
- **Task 1c `fit_gao_bon(d, gold_gain)`**: least-squares fit of $d(\alpha - \beta d)$.

**Parts 2–3, `failures.py`**

- **Task 2 `kl_penalized_rewards`**: $r_i - \beta \sum_t (\log\pi - \log\pi_\text{ref})$ over the real (masked) tokens.
- **Task 3 `clipped_pg_loss`**: the PPO/GRPO clipped surrogate with separate $\epsilon_\text{low}$ and $\epsilon_\text{high}$, token-level mean. Tests check value *and* gradient in six clip cases.
- **Task 4 `token_entropy`**: mean per-token entropy over response positions.

```bash
python -m pytest -q                   # all tests; they fail until you fill in the TODOs
python -m pytest -q -k task3          # just Task 3
RL_SOLUTIONS=1 python -m pytest -q    # the reference solutions: all pass
```

## Then run the experiments

```bash
python run.py overopt    # a few seconds
python run.py hack       # ~2 minutes; the first run also trains and caches sft_tinylm.pt (~10 s)
python run.py entropy    # ~2 minutes
python run.py            # all three (~5 minutes)
```

**Before Part 2, make a prediction.** Read `proxy_reward` in `common.py`: full credit if the correct answer appears *anywhere* in the response, plus 0.1 per character. What will the policy learn? What will happen to exact-match accuracy?

**What you should see** (reference solution; your numbers can differ a little):

- **Part 1.** With a length-biased proxy ($b = 1$), the gold gain peaks at best-of-10 (KL ≈ 1.4 nats, gain ≈ +0.66) and is negative by best-of-3000, while the proxy keeps rising. The fit gives $\alpha \approx 1.1$, $\beta \approx 0.47$. With a noise-only proxy, $\beta \approx 0$: the gold never turns down. Don't over-read the far right of each curve: at $n$ in the thousands the winner comes from the top few hundred of the 200,000 pooled responses, so those points are noisy (for $b = \sigma = 1$ the pool says about −0.4 at $n = 3000$; the exact value is about +0.10).
- **Part 2.** *Proxy, no KL*: accuracy first rises (≈ 0.52 → 0.62 around step 50), then the policy learns to keep writing digits after the answer. Length jumps to the 4-character maximum, KL to ≈ 15 nats, and accuracy falls to 0 while the proxy keeps climbing. *Proxy with β = 0.05*: no hack; accuracy rises to ≈ 0.75, close to the *gold reward* run (≈ 0.8).
- **Part 3.** Entropy falls from ≈ 0.35 to ≈ 0.15 nats per token within 120 steps with either clip, and the symmetric clip and clip-higher are nearly indistinguishable at this scale (run a few seeds before believing any gap). With no upper clip, entropy stays higher (≈ 0.19) but accuracy falls: the run is unstable. The diagnostic is the clearest signal: the upper clip switches off over 90% of the *rare* positive-advantage tokens and almost none of the *common* ones.

## Things to try

- Part 2 with `LENGTH_BONUS = 0.0` in `common.py`, or with an exact checker plus the length bonus. Is either flaw exploitable on its own?
- Part 2 with β = 0.01 and β = 0.02. Where does the leash stop working?
- Part 2 with std-normalized advantages (divide by the group std, as in the original GRPO). Does the hack arrive sooner? (Think about a group where every response is correct and one is padded.)
- Part 1 with another pool seed or a 2,000,000-response pool (edit `make_pool(...)` in `run.py`). Which parts of the curves move?
- Part 3 with more seeds (`RLConfig(seed=...)`), `epochs=2` (does clip-higher matter more when ratios overshoot less?), or ε_high = 0.4.