Paper map

Every paper, book and post the chapters recommend, in one place. Start with the reading path, then use the chapter-by-chapter list when you want to go deeper on one topic.

How to use this map

Every chapter ends with a Further reading list. This page collects all of them, with duplicates removed, so you can see the literature behind the whole course at once.

• New to the field? Follow A reading path below. It is a short list, in order, that takes you from the first ideas of RL to how language models are trained with RL today. Read each paper after the chapter it is linked to: the chapter gives you the notation and intuition, and the paper then reads much faster.
• Studying one chapter? Jump to that chapter in All papers by chapter. Each paper is listed once, under the chapter that uses it most; “Also in” tells you where else it comes up.
• Don’t read papers cover to cover. On a first pass, read the abstract, the main figure, and the section that defines the method. Each entry tries to tell you which part to read.

A small exercise that works for every paper: after reading, write the paper’s objective (the loss or the expected return it maximizes) in this course’s notation, and say in one sentence which earlier method it changes and why. If you can do that, you have understood the paper well enough to use it.

A reading path

Thirteen readings, in order. Together they cover the main line of the course: values, then policy gradients, then PPO, then RLHF, then reasoning models.

1. Sutton & Barto (2018), Reinforcement Learning: An Introduction. The free textbook for all of Part I: agents, rewards, MDPs, value functions and TD learning. Read chapter 1 first. (Ch. 1)
2. Mnih et al. (2015), Human-level control through deep reinforcement learning. DQN: the first time a neural network learned values well enough to play Atari from pixels. (Ch. 6)
3. Williams (1992), Simple statistical gradient-following algorithms for connectionist reinforcement learning. REINFORCE, the policy-gradient estimator that GRPO and its relatives still use. (Ch. 7)
4. Schulman, Moritz, Levine, Jordan & Abbeel (2015), High-Dimensional Continuous Control Using Generalized Advantage Estimation. GAE: how a learned critic turns noisy returns into usable advantages. (Ch. 8)
5. Schulman et al. (2017), Proximal Policy Optimization Algorithms. PPO: the clipped objective behind RLHF and most LLM RL since. (Ch. 9)
6. Christiano et al. (2017), Deep reinforcement learning from human preferences. Learning a reward model from pairwise human comparisons; the idea RLHF is built on. (Ch. 12)
7. Ouyang et al. (2022), Training language models to follow instructions with human feedback. InstructGPT: the SFT, reward model, PPO recipe that made chat assistants. (Ch. 12)
8. Gao, Schulman & Hilton (2022), Scaling Laws for Reward Model Overoptimization. What happens when you optimize hard against a learned reward: the true score rises, then falls. (Ch. 19)
9. Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model. DPO: the reward model and PPO folded into one supervised loss. (Ch. 13)
10. Shao et al. (2024), DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Where GRPO comes from: PPO without a critic, using a group of samples as the baseline. (Ch. 15)
11. DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. RL with verifiable rewards producing a reasoning model, including R1-Zero trained with RL alone. (Ch. 15)
12. Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Four practical fixes to GRPO, each with an ablation; a good picture of what tuning LLM RL looks like. (Ch. 16)
13. Team Olmo (2025), Olmo 3. A fully open RLVR recipe with data, code and logs; the best place to start if you want to reproduce frontier-style RL yourself. (Ch. 21)

All papers by chapter

Each reading appears once. “Also in” lists the other chapters that recommend it, often for a different section.

Part I: Foundations

Chapter 1: What is reinforcement learning?

• Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed. The standard textbook, free online. For this chapter: chapter 1 and sections 3.1–3.2 (the reward hypothesis). Also in Ch. 2 (book ch. 2), Ch. 3 (ch. 3), Ch. 4 (ch. 4), Ch. 5 (ch. 5–6), Ch. 6 (ch. 9–11), Ch. 7 (ch. 13), Ch. 8 (sections 13.4–13.5, ch. 7 and 12), Ch. 10 (ch. 8).
• OpenAI Spinning Up (2018), Part 1: Key Concepts in RL. A compact tour of the same vocabulary in deep-RL notation; useful as a cheat sheet later. Also in Ch. 3.
• Clark & Amodei (2016), Faulty reward functions in the wild. The CoastRunners boat that learned to spin in circles instead of finishing the race, with video.
• Bowling, Martin, Abel & Dabney (2023), Settling the reward hypothesis. Exactly when a single scalar reward is enough to express a goal.

Chapter 2: Bandits

• Lattimore & Szepesvári (2020), Bandit Algorithms. The reference for bandit theory, free online. Read the chapters on UCB and on lower bounds first.
• Russo, Van Roy, Kazerouni, Osband & Wen (2018), A Tutorial on Thompson Sampling. A readable tutorial with many examples.
• Auer, Cesa-Bianchi & Fischer (2002), Finite-time Analysis of the Multiarmed Bandit Problem. The UCB1 paper; its Theorem 1 is the proof sketched in the chapter.
• Slivkins (2019), Introduction to Multi-Armed Bandits. A shorter, more lecture-like book than Lattimore & Szepesvári.
• Li, Chu, Langford & Schapire (2010), A Contextual-Bandit Approach to Personalized News Article Recommendation. LinUCB, and contextual bandits running on a real website.
• Kaufmann, Korda & Munos (2012), Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis. Thompson sampling matches the Lai–Robbins lower bound for Bernoulli arms.

Chapter 3: Markov decision processes

• David Silver, UCL Course on RL. Recorded lectures with slides. Lecture 2 is MDPs; lectures 4 and 5 (model-free prediction and control) go with Ch. 5. Also in Ch. 4, which links lecture 3 (dynamic programming) at this copy of the course page.
• Kaelbling, Littman & Cassandra (1998), Planning and acting in partially observable stochastic domains. The standard introduction to POMDPs and belief states.

Chapter 4: Bellman equations and dynamic programming

• Puterman (1994), Markov Decision Processes: Discrete Stochastic Dynamic Programming. The rigorous reference for MDP theory, including policy iteration as Newton’s method. Also in Ch. 3.
• Agarwal, Jiang, Kakade & Sun (2019–2022), Reinforcement Learning: Theory and Algorithms. Free book; chapter 1 proves the chapter’s results with convergence rates.
• Bertsekas (2019), Reinforcement Learning and Optimal Control. DP from the control-theory side, with a bridge to approximate methods (Chapters 6–9).
• Szepesvári (2010), Algorithms for Reinforcement Learning. A short, tidy monograph; its opening chapter covers Bellman operators and contraction in a few pages.

Chapter 5: Monte Carlo and TD learning

• Sutton (1988), Learning to predict by the methods of temporal differences. The paper that introduced TD learning.
• Watkins & Dayan (1992), Q-learning. The short convergence proof for tabular Q-learning.
• van Hasselt (2010), Double Q-learning. Maximization bias and the double estimator.
• Tesauro (1995), Temporal difference learning and TD-Gammon. The first big TD success: near-expert backgammon by self-play.

Part II: Deep RL

Chapter 6: Function approximation and DQN

• Mnih et al. (2015), Human-level control through deep reinforcement learning. The Nature DQN paper. Read the Methods section and the replay/target-network ablation. Also in Ch. 1.
• Mnih et al. (2013), Playing Atari with Deep Reinforcement Learning. The short workshop paper that started it.
• van Hasselt, Guez & Silver (2016), Deep Reinforcement Learning with Double Q-learning. Overestimation and the one-line fix.
• Hessel et al. (2018), Rainbow: Combining Improvements in Deep Reinforcement Learning. Six DQN extensions with an ablation of each.
• van Hasselt et al. (2018), Deep Reinforcement Learning and the Deadly Triad. When deep Q-learning actually diverges.
• Tsitsiklis & Van Roy (1997), An analysis of temporal-difference learning with function approximation. The convergence proof for on-policy linear TD, and divergence examples.
• Wang et al. (2016), Dueling Network Architectures for Deep Reinforcement Learning, and Schaul et al. (2016), Prioritized Experience Replay. Two of Rainbow’s ingredients, each short.

Chapter 7: Policy gradients and REINFORCE

• Williams (1992), Simple statistical gradient-following algorithms for connectionist reinforcement learning. The original REINFORCE paper, including the baseline idea.
• OpenAI Spinning Up, Part 3: Intro to Policy Optimization. A short, careful derivation with code, including reward-to-go.
• Sutton, McAllester, Singh & Mansour (1999), Policy Gradient Methods for Reinforcement Learning with Function Approximation. The policy gradient theorem and the bridge to actor-critic.
• Greensmith, Bartlett & Baxter (2004), Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. Baselines as control variates, with optimal baselines.
• Mohamed, Rosca, Figurnov & Mnih (2020), Monte Carlo Gradient Estimation in Machine Learning. Where REINFORCE sits among gradient estimators.
• Nota & Thomas (2020), Is the Policy Gradient a Gradient? What the dropped $\gamma^t$ factor means for what your code optimizes.

Chapter 8: Actor-critic and advantage estimation

• Schulman, Moritz, Levine, Jordan & Abbeel (2015), High-Dimensional Continuous Control Using Generalized Advantage Estimation. The GAE paper. Sections 2–3 match the chapter closely.
• Mnih et al. (2016), Asynchronous Methods for Deep Reinforcement Learning. A3C: parallel actors, n-step advantages and an entropy bonus.
• Pardo, Tavakoli, Levdik & Kormushev (2018), Time Limits in Reinforcement Learning. Why a time-out is not a terminal state.
• Barto, Sutton & Anderson (1983), Neuronlike adaptive elements that can solve difficult learning control problems. The original actor-critic, balancing a pole.

Chapter 9: Trust regions, TRPO and PPO

• Schulman et al. (2017), Proximal Policy Optimization Algorithms. Short and readable. Section 3 is the clipped objective.
• Schulman et al. (2015), Trust Region Policy Optimization. The theory: monotonic improvement, the KL constraint, conjugate gradient.
• Huang et al. (2022), The 37 Implementation Details of Proximal Policy Optimization. Every trick in real PPO code. Read it before you write PPO for anything serious. Also in Ch. 8 (for GAE and advantage normalization).
• OpenAI Spinning Up, PPO and TRPO. Compact explanations with pseudocode and clean implementations.
• Engstrom et al. (2020), Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. Code-level details drive much of PPO’s performance.
• Andrychowicz et al. (2021), What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study. Hundreds of thousands of runs measuring which on-policy choices matter. Also in Ch. 8 (dated 2020 there, the arXiv year).

Chapter 10: The wider RL landscape

• Browne et al. (2012), A Survey of Monte Carlo Tree Search Methods. The standard MCTS reference.
• Silver et al. (2017), Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. AlphaZero: search plus a network trained to imitate the search.
• Schrittwieser et al. (2019), Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. MuZero: search inside a learned model.
• Ross, Gordon & Bagnell (2011), A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. DAgger, and why behavior cloning’s errors compound.
• Levine, Kumar, Tucker & Fu (2020), Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Why offline RL is hard, and the main fixes.
• Levine (2018), Reinforcement Learning and Control as Probabilistic Inference. Maximum-entropy RL and the Boltzmann optimum; the bridge to the KL-regularized view of RLHF.

Part III: RL for LLMs

Chapter 11: The language model as a policy

• Schulman (2020), Approximating KL Divergence. A two-page note defining the $k_1$, $k_2$, $k_3$ KL estimators used everywhere. Also in Ch. 9 and Ch. 15 (the estimator GRPO uses).
• Rafailov, Hejna, Park & Finn (2024), From r to Q*: Your Language Model is Secretly a Q-Function. The token-level MDP view of the KL-regularized objective, connected to DPO.
• Korbak, Perez & Buckley (2022), RL with KL penalties is better viewed as Bayesian inference. The optimal policy as a posterior with the reference model as prior.
• Shah et al. (2025), A Comedy of Estimators: On KL Regularization in RL Training of LLMs, and Tang & Munos (2025), On a few pitfalls in KL divergence gradient estimation for RL. Where the KL goes and which estimator you differentiate change the gradient you actually get.
• Holtzman et al. (2020), The Curious Case of Neural Text Degeneration. Why greedy and pure sampling both fail, and nucleus (top-p) sampling.
• Lambert, Reinforcement Learning from Human Feedback (online book). A practitioner’s tour of the whole post-training pipeline; read alongside Part III. Also in Ch. 12.

Chapter 12: RLHF, reward models and PPO

• Ouyang et al. (2022), Training language models to follow instructions with human feedback. InstructGPT, the canonical three-step recipe. Section 3 is the method; Appendix C has the hyperparameters. Also in Ch. 1 (skim Figure 2) and Ch. 11 (section 3.5, the KL-penalized objective).
• Christiano et al. (2017), Deep reinforcement learning from human preferences. Reward models from pairwise comparisons, on Atari and robot tasks. Also in Ch. 10.
• Stiennon et al. (2020), Learning to summarize from human feedback. Where RLHF on language models clearly worked, with a reward-overoptimization figure.
• Huang et al. (2024), The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. Every small detail of an open PPO-RLHF reproduction.
• Ziegler et al. (2019), Fine-Tuning Language Models from Human Preferences. The KL-in-the-reward formulation and the adaptive KL controller.
• Bai et al. (2022a), Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Anthropic’s version, with iterated online RLHF and the helpful–harmless tension.
• Bradley & Terry (1952), Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. The original paired-comparison model, from statistics.

Chapter 13: Direct preference optimization

• Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The original paper. Section 4 is the derivation.
• Azar et al. (2023), A General Theoretical Paradigm to Understand Learning from Human Preferences. IPO, and why deterministic preferences break DPO’s regularization.
• Razin et al. (2025), Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. Why the chosen response’s probability can fall during DPO.
• Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. A theorem on DPO’s solutions, and the PPO tuning that made it win.
• Ivison et al. (2024), Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. A controlled comparison; data quality matters most.
• Meng, Xia & Chen (2024), SimPO: Simple Preference Optimization with a Reference-Free Reward, and Ethayarajh et al. (2024), KTO: Model Alignment as Prospect Theoretic Optimization. The two most widely used DPO variants.
• Guo et al. (2024), Direct Language Model Alignment from Online AI Feedback, and Tajwar et al. (2024), Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. Why on-policy data matters, from two angles.

Chapter 14: Where rewards come from

• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The reference study of LLM judges and their biases.
• Bai et al. (2022b), Constitutional AI: Harmlessness from AI Feedback. Critique–revision and RL from AI preferences. Read sections 3 and 4.
• Lee et al. (2023), RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. A careful head-to-head comparison, including position bias.
• Gunjal et al. (2025), Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. Rubric aggregation formulas and worked examples.
• Malik et al. (2025), RewardBench 2: Advancing Reward Model Evaluation. How reward models are evaluated today. Pair with the original RewardBench.
• Eisenstein et al. (2023), Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. What reward-model ensembles can and can’t fix.

Chapter 15: RL with verifiable rewards, reasoning models and GRPO

• Shao et al. (2024), DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Where GRPO comes from. Section 4 has the objective and a “unified paradigm” of SFT, DPO, PPO and GRPO. Also in Ch. 16.
• DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, revised in Nature 645, 633–638. R1-Zero, the R1 pipeline, and an honest “unsuccessful attempts” section.
• Yue et al. (2025), Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? The pass@k study behind the “sharpening” view. Read it alongside ProRL (Liu et al., 2025) for the other side. Also in Ch. 21.
• Kimi Team (2025), Kimi k1.5: Scaling Reinforcement Learning with LLMs. The parallel recipe from the same week, with partial rollouts for long contexts.
• Lambert et al. (2024), Tülu 3: Pushing Frontiers in Open Language Model Post-Training. The fully open recipe that named RLVR. Also in Ch. 13 (its DPO stage) and Ch. 14 (its verifiers).
• Hugging Face TRL, GRPO Trainer documentation. For running GRPO on a real model: reward-function conventions, loss variants and logged metrics.

Chapter 16: The critic-free algorithm zoo

• Liu et al. (2025), Understanding R1-Zero-Like Training: A Critical Perspective. Dr. GRPO: GRPO’s length and difficulty biases, and a critical look at the “aha moment”. Also in Ch. 15.
• Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Four fixes and an ablation table that prices each one.
• Ahmadian et al. (2024), Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. RLOO, and the argument that PPO’s machinery is mostly unnecessary for LLMs. Also in Ch. 7.
• Zheng et al. (2025), Group Sequence Policy Optimization. GSPO: sequence-level importance ratios.
• MiniMax (2025), MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. CISPO, plus the FP32 LM-head fix for train–inference mismatch.
• Khatri et al. (2025), The Art of Scaling Reinforcement Learning Compute for LLMs. ScaleRL: which knobs change the final performance and which only change the speed. Also in Ch. 20 and Ch. 21.
• Hu et al. (2025), REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Normalization. Batch-level rather than per-prompt normalization.

Chapter 17: Process rewards and test-time compute

• Snell, Lee, Xu & Kumar (2024), Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. A map of test-time strategies and when each wins. Sections 5–7.
• Lightman et al. (2023), Let’s Verify Step by Step. PRM800K: process vs. outcome supervision. Short and readable.
• Wang et al. (2024), Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. Process labels from Monte Carlo rollouts.
• Cobbe et al. (2021), Training Verifiers to Solve Math Word Problems. GSM8K and the original outcome reward model.
• Wang et al. (2022), Self-Consistency Improves Chain of Thought Reasoning in Language Models. Majority voting, the baseline everyone compares against.
• Setlur et al. (2024), Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. Process rewards as advantages under a prover policy.
• Cui et al. (2025), Process Reinforcement through Implicit Rewards (PRIME), with Yuan et al. (2024), Free Process Rewards without Process Labels. Dense rewards from an implicit PRM.
• Muennighoff et al. (2025), s1: Simple test-time scaling. Budget forcing. Also read Stroebl, Kapoor & Narayanan, The Limits of Inference Scaling Through Resampling, on why imperfect verifiers cap resampling.

Chapter 18: Agentic and multi-turn RL

• Jin et al. (2025), Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. The cleanest small-scale agentic RL paper.
• Agentica & Together AI (2025), DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. An honest recipe for RL on real code repositories.
• Wang et al. (2025), RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. Multi-turn RL on small environments and the “Echo Trap”.
• Feng et al. (2025), Group-in-Group Policy Optimization for LLM Agent Training. Step-level credit without a critic.
• Wei et al. (2025), SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. A similarity reward with no code execution.
• GLM-4.5 Team (2025), GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. A production report that spells out agentic RL choices.
• Zhang et al. (2025), The Landscape of Agentic Reinforcement Learning for LLMs: A Survey. A map of the field, for breadth.

Chapter 19: Failure modes

• Gao, Schulman & Hilton (2022), Scaling Laws for Reward Model Overoptimization. The paper behind the chapter’s central curve. Sections 3–4.
• Weng (2024), Reward Hacking in Reinforcement Learning. A long, well-organized survey with many references.
• Krakovna et al. (2020), Specification gaming: the flip side of AI ingenuity. A short essay with a link to a public list of examples. Also in Ch. 1.
• Baker et al. (2025), Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. Real reward hacks in frontier coding RL, and what happens when you train against the monitor.
• MacDiarmid et al. (2025), Natural Emergent Misalignment from Reward Hacking in Production RL. Why hack rates are a safety metric, and inoculation prompting. Also in Ch. 18 (read before you build an environment) and Ch. 21.
• Cui et al. (2025), The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. Entropy collapse, the covariance rule and Clip-Cov/KL-Cov.
• Sharma et al. (2023), Towards Understanding Sycophancy in Language Models. Evidence that human preference data rewards telling people what they want to hear. Also in Ch. 14.
• Pan, Bhatia & Steinhardt (2022), The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. Phase transitions in reward hacking, in classic RL environments.

Chapter 20: RL systems for LLMs

• Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention. The vLLM paper; why decoding is memory-bound.
• Sheng et al. (2024), HybridFlow: A Flexible and Efficient RLHF Framework. The verl paper: the dataflow view of an RL step.
• Fu et al. (2025), AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. Asynchronous RL with staleness control.
• Noukhovitch et al. (2024), Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. How much off-policy data LLM RL can tolerate.
• Yao et al. (2025), Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. The blog post that named the train–inference mismatch.
• He and Thinking Machines (2025), Defeating Nondeterminism in LLM Inference. Why inference is nondeterministic, and what bitwise on-policy RL buys.
• Schulman and Thinking Machines (2025), LoRA Without Regret. LoRA for SFT and RL, with practical learning-rate advice.

Chapter 21: The frontier

• DeepSeek-AI (2026), DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, section 5. A detailed public description of a 2026 frontier post-training pipeline.
• Lu and Thinking Machines (2025), On-Policy Distillation. The most readable introduction to on-policy distillation as RL.
• Tajwar et al. (2026), Maximum Likelihood Reinforcement Learning. One view that unifies REINFORCE, GRPO and maximum likelihood. Sections 3–5.
• Team Olmo (2025), Olmo 3. A fully open RLVR recipe with data, code and logs.
• Korbak et al. (2025), Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. A short cross-lab position paper on keeping reasoning readable.