# Lab 14: Where rewards come from

The reward function is the part of an RL pipeline that people write by hand,
and it is where most of the bugs hide. In this lab you build three real pieces
of reward plumbing from Chapter 14:

1. **A math-answer verifier** (Tasks 1–3): the reward function of RLVR. Extract
   the final answer, normalize it, compare it with the gold answer, and refuse
   to reward hedging. Most of the work is in the tricky cases, and the tests
   contain about 80 of them.
2. **A rubric aggregator** (Task 4): turn per-criterion judgments into one
   scalar reward, for tasks that have no answer key.
3. **Order swapping for an LLM judge** (Task 5): cancel position bias, and
   see for yourself that it does nothing about verbosity bias.

Everything is plain Python plus numpy (for `run.py`). No sympy and no `eval`:
the point is to know exactly what your verifier accepts. The tests run in
under a second; `run.py` takes about two seconds.

## Files

| File | What it is |
|---|---|
| `rewards.py` | **Your file.** Five tasks marked `TODO`. |
| `helpers.py` | Complete helper code: `normalize_text`, two naive verifiers, the `SimulatedJudge`, and the answer "strategies" used by `run.py`. Read it once. |
| `test_rewards.py` | The tests. |
| `run.py` | Uses your code: prints a verifier gallery, trains a tiny policy against a naive verifier and against yours, and plots judge-bias curves to `reward_sources.png`. |
| `solutions/rewards.py` | Reference solution. Try not to peek. |

## Tasks (in order)

| Task | Function | Chapter section | Test with |
|---|---|---|---|
| 1 | `extract_boxed`: every `\boxed{...}`, with brace matching | Verifiers: rewards you can compute | `pytest -q -k extract_boxed` |
| 2 | `parse_number`: `0.5`, `1/2`, `\frac{1}{2}`, `1,000`, `x = 5`, ... to an exact `Fraction` | Verifiers | `pytest -q -k parse_number` |
| 3 | `math_reward`: the verifier, with an anti-hedging rule | Verifiers, and Exercise 14.6 | `pytest -q -k math_reward` |
| 4 | `rubric_reward`: weighted criteria to a scalar in [0, 1] | Rubrics: a checklist instead of a vibe | `pytest -q -k rubric` |
| 5 | `debiased_preference` and `consistent_verdict` | LLM-as-a-judge, and Exercise 14.4 | `pytest -q -k "debiased or verdict or swapping"` |

Tips:

- **Task 1.** Don't use the regex `\\boxed\{(.*?)\}`: it stops at the first `}`
  and returns `\frac{1` for `\boxed{\frac{1}{2}}`. Walk the string and count
  brace depth instead. A backslash escapes the next character.
- **Task 2.** Follow the numbered steps in the docstring, in order. Use
  `fractions.Fraction` (`Fraction("0.125") == Fraction(1, 8)` exactly), never
  floats: `0.1 + 0.2 != 0.3`. When a test fails, the message shows the input.
  Print your string after each step to see where it goes wrong.
- **Task 3.** Two candidates are "the same answer" if both parse to the same
  number, or (if either one is not a number) their `normalize_text` forms are
  equal. Stop counting distinct answers as soon as you exceed `max_answers`:
  the "spam" response boxes 179 numbers.
- **Task 5.** The judge returns P(first shown is better). So P(a wins) when `b`
  is shown first is `1 - judge(prompt, b, a)`.

## Running

```bash
python -m pytest -q                  # fails until you fill in the TODOs
RL_SOLUTIONS=1 python -m pytest -q   # the reference solution: all pass
python run.py                        # your code; saves reward_sources.png
```

## What you should see

`run.py` prints something like:

```text
1. Verifier gallery (1 = rewarded)
   case                                     substring  last num  yours
   honest and correct                               1         1      1
   same value, LaTeX form                           0         0      1
   thousands separator                              0         0      1
   hedging with two answers                         1         1      0
   spam every number                                1         0      0
   wrong, but contains the gold string              1         0      0
   correct, then a stray number                     1         0      1
   correct, no box                                  1         1      1

2. Training a 4-strategy policy against two verifiers (300 steps x 32 samples each)...
   final policy vs naive substring verifier : honest (boxed) 0.01, honest (unboxed) 0.01, hedge 3 answers 0.55, spam all answers 0.43
   final policy vs your math_reward         : honest (boxed) 0.67, honest (unboxed) 0.32, hedge 3 answers 0.01, spam all answers 0.01

3. Judge bias: P(judge prefers the better answer)
   position bias b=3, equal lengths: shown second 0.12, swapped 0.55
   verbosity bias 1.0 vs a 4x longer, 1-point-worse answer, swapped: 0.31
```

- **Gallery.** Each naive verifier has *false negatives* (it misses correct
  answers written differently) and *false positives* (it pays for wrong ones).
  Yours should be all ones on the correct rows and zeros on the cheating rows.
- **Top panels.** The "policy" chooses between four ways of answering an
  addition problem; honest answers are right 60% of the time. Against the
  substring verifier, hedging and spamming earn reward 1.0 every time, so
  REINFORCE drives the honest strategies down to a few percent within about 100 steps.
  Nobody told the policy to cheat: optimization found the hole. Against your
  verifier, hedging and spamming earn 0, and the policy stays honest. (It
  splits between boxed and unboxed honest answers because your fallback gives
  them the same reward. Exercise: make the unboxed one worth 0.9 and watch.)
- **Bottom left.** With equal lengths, a position bias of b ≈ 1 or more makes
  the better answer lose whenever it is shown second. Averaging the two orders
  always keeps the right verdict (the curve stays above 0.5), although the
  margin shrinks toward 0.5 as b grows.
- **Bottom right.** Against a 4× longer, 1-point-worse answer, the debiased
  judge prefers the padded answer once λ > 0.5 logits per doubling. Swapping
  can't help: the padded answer is longer in both orders.

## Going further (optional)

- Add `\sqrt{...}` and `\pi` support to `parse_number` without using `eval`.
  (Hint: you'll need to compare with a tolerance, and then think about what
  tolerance a policy could exploit.)
- Make `math_reward` reject responses longer than some limit, and add a
  "format reward" of 0.1 for using `\boxed{}`, as in DeepSeek-R1-Zero. Retrain
  the strategy policy in `run.py`. Which strategy wins now?
- Write a `length_controlled_preference` that fits the verbosity coefficient λ
  from many judgments (logistic regression on log2 length ratio) and subtracts
  it. How many judgments do you need to estimate λ to ±0.1?