A self-paced course · 21 chapters · exercises with solutions
Reinforcement learning,
from scratch to LLMs
Start with a robot in a grid and a row of slot machines. End with how GRPO, verifiable rewards and agentic RL post-train today's language models. Every chapter has a demo you can play with, exercises to work through (pen and paper, and code with tests), and solutions you can open when you're ready.
Part I
Foundations
Agents, rewards, values. Everything here fits in a table — no neural networks yet.
- 01What is reinforcement learning?Agent, environment, action, reward. How learning from reward differs from learning from labels.
- 02Bandits: exploration vs. exploitationThe one-state RL problem. ε-greedy, UCB, Thompson sampling, and regret.
- 03Markov decision processesStates, transitions, returns, discounting, policies, and value functions.
- 04Bellman equations and dynamic programmingSolving an MDP when you know the rules: policy evaluation, policy iteration, value iteration.
- 05Learning from experience: Monte Carlo and TDLearning values without a model. TD(0), SARSA, Q-learning, and on- vs. off-policy.
Part II
Deep RL
Neural networks as value functions and policies, and the policy-gradient family that LLM training is built on.
- 06Function approximation and DQNWhen the table is too big: neural Q-functions, experience replay, target networks.
- 07Policy gradients and REINFORCEOptimize the policy directly. The log-derivative trick, baselines, and variance.
- 08Actor-critic and advantage estimationA learned critic to cut variance. Advantages, n-step returns, and GAE.
- 09Trust regions: TRPO and PPOTake big, safe policy steps. Importance ratios, clipping, and the algorithm behind RLHF.
- 10The wider RL landscapeModel-based RL and AlphaZero/MuZero, offline RL, exploration, and where each idea resurfaces in LLMs.
Part III
RL for LLMs
How RL turns a pretrained language model into an assistant and a reasoner: RLHF, DPO, GRPO, verifiable rewards, agents, and the current frontier.
- 11The language model as a policyTokens as actions, prompts as states. Pretraining → SFT → RL, and why a KL leash to the reference model.
- 12RLHF: reward models and PPOThe InstructGPT recipe: preference data, a Bradley–Terry reward model, and PPO with a KL penalty.
- 13Direct preference optimizationSkip the reward model: derive DPO from the KL-regularized objective, and meet its variants.
- 14Where rewards come fromHuman labels, AI feedback, Constitutional AI, LLM judges, rubrics, and verifiers.
- 15RL with verifiable rewards: reasoning models and GRPOo1 and DeepSeek-R1. Group-relative advantages, R1-Zero, and why checkable answers changed everything.
- 16The critic-free algorithm zooRLOO, GRPO, Dr. GRPO, DAPO, GSPO, CISPO and friends — one estimator, different knobs.
- 17Process rewards and test-time computeRewarding steps vs. outcomes, PRMs, search, and trading inference compute for accuracy.
- 18Agentic and multi-turn RLTool use, coding agents, environments, and credit assignment over long horizons.
- 19Failure modes: reward hacking and friendsOver-optimization, reward hacking, length bias, sycophancy, and entropy collapse — and the fixes.
- 20RL systems for LLMsWhy rollouts dominate cost, async and off-policy training, train–inference mismatch, and the open-source stacks.
- 21The frontier (as of October 2026)What the latest models do in post-training, and the open problems worth watching.
How to use this course
- Read with the demo open. Each idea comes with something to poke at. Change a slider before you read the explanation of what it does.
- Do the exercises before opening solutions. Pen-and-paper ones build the math; code ones come with tests so you know when you're right.
- Follow the arrows to Part III. Boxes marked “Where this shows up in LLMs” connect each classic idea to modern post-training.
- Your progress stays in this browser. Tick off exercises and chapters as you go.