DeepSeek-R1: How Pure RL Taught a Model to Reason
DeepSeek-R1 showed that a language model can learn long, self-checking chains of thought when it is rewarded only for getting the final answer right, with no human-written reasoning to copy. This paper insight walks through the paper section by section, equations included, so you can explain what was actually done instead of repeating the headlines: GRPO and why it drops PPO's value model, the accuracy and format rewards, the emergent reflection behind the 'aha moment', and the multi-stage pipeline that turned a raw experiment into a usable assistant.
The course is five short lessons on a prerequisite path. An AI tutor teaches each one with narrated slides and then quizzes you on it, so you move on only once the idea has stuck. You will compute group-relative advantages by hand, read the GRPO objective term by term, work out a language-consistency reward, and weigh the paper's own evidence on distillation versus reinforcement learning.
It suits engineers and students who already know roughly what pre-training and fine-tuning are and want to understand reasoning models from the primary source. It also covers what the authors say did not work, including process reward models and tree search, and the limits they report, so you finish with a realistic picture of where pure RL helps and where it stops.
Estimated effort: 4 hours
What you will learn
- Compute GRPO's group-relative advantages and explain each term of the GRPO objective, including the clip and the KL penalty
- Explain why GRPO needs no value model and when a tuned PPO comes close to it
- Score responses under DeepSeek-R1-Zero's accuracy and format rewards and explain why neural reward models were avoided for reasoning
- Trace the four stages of the DeepSeek-R1 pipeline, from cold-start data to the final RL stage with helpfulness and safety reward models
- Compare distilling DeepSeek-R1 into small models with running RL on them, using the paper's results
- Describe the failed process-reward-model and MCTS attempts and the limits the paper reports, such as reward hacking and overthinking
Course syllabus
- Why DeepSeek-R1 Matters: The Question, the Two Models, and the Roadmap
- The question the paper asks: can a language model get better at hard reasoning if we only tell it whether its final answer was right, without ever showing it how to think?
- Why that question matters: worked solutions written by people are slow and expensive to collect, and they may cap a model at the way people happen to solve problems
- The two models in the paper: a first experiment trained only by being rewarded for right answers, and a polished model built on top of what that experiment taught
- What surprised the authors: the experimental model started writing longer answers and checking its own work without being told to
- How the rest of the paper turns that discovery into a readable, general-purpose assistant and then into small models anyone can run
- Roadmap: the learning algorithm, the first experiment and what emerged, the full training pipeline, and finally shrinking the model and the limits the authors admit
- GRPO: Group Relative Policy Optimization and Why It Drops PPO's Value Model
- Vocabulary for an LLM as an RL agent: the policy is the model, an action is the generated output, the reward scores the output, and the advantage says how much better than expected an output was
- The GRPO recipe: for each question q, sample a group of G outputs from the old policy, score each one, and compute A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G) (paper Eq. 3)
- The GRPO objective (Eq. 1) read term by term: the probability ratio pi_theta(o_i|q) / pi_theta_old(o_i|q), the min with a clipped ratio in [1 - epsilon, 1 + epsilon], averaging over the group, and minus beta times the KL divergence to a reference policy
- The KL estimator (Eq. 2): ratio pi_ref / pi_theta minus log of that ratio minus 1, which is always non-negative, and why the reference policy is refreshed to the latest policy every 400 steps
- PPO for contrast: a value model of similar size to the policy plus Generalized Advantage Estimation (GAE); its memory and compute overhead; why predicting the final reward from a partial chain of thought is especially hard when later text revises earlier text
- Why a per-token KL reward (as in PPO-based RLHF) implicitly penalizes long responses, while GRPO's loss-level KL does not block response length from growing
- Evidence in the paper (Supplementary A.3): PPO with the default GAE lambda of 0.95 trails GRPO on MATH; tuned to lambda = 1.0 it nearly matches, at extra tuning cost
- DeepSeek-R1-Zero's RL settings: learning rate 3e-6, KL coefficient 0.001, temperature 1, 16 outputs per question, 32 questions per step (batch 512), max length 32,768 tokens raised to 65,536 after step 8.2k, 10,400 steps in total
- DeepSeek-R1-Zero: Rule-Based Rewards, a Minimal Template, and the 'Aha Moment'
- The starting point: DeepSeek-V3-Base, with no supervised fine-tuning before RL, because human-written reasoning patterns might limit what the model explores
- The training template (Table 1): reasoning inside think tags followed by the answer inside answer tags, and deliberately nothing about how to reason
- Accuracy reward: deterministic checks such as an answer in a specified box for math or a compiler and test cases for code
- Format reward: reasoning must sit between the think tags; Reward_rule = Reward_acc + Reward_format with equal weight (Eq. 4)
- Why no neural reward models (outcome- or process-based) for reasoning: they are open to reward hacking at large scale, and retraining them adds compute and complexity
- Results: AIME 2024 pass@1 rose from 15.6% to 77.9%, and 86.7% with self-consistency over 16 samples, beyond the average human competitor
- Self-evolution: average response length climbs steadily to thousands of tokens; reflective words such as 'wait', 'verify', 'mistake' rise 5 to 7 times; 'wait' is almost absent early and spikes after about step 8,000
- The 'aha moment' (Table 2): an intermediate checkpoint stops mid-derivation, flags its own error and re-evaluates step by step
- What R1-Zero got wrong: poor readability, mixing English and Chinese words within one chain of thought, and weak results outside verifiable reasoning such as writing
- The DeepSeek-R1 Pipeline: Cold Start, Two RL Stages, and Rejection-Sampled SFT
- Why a second model: fix R1-Zero's readability and language mixing and extend it beyond verifiable reasoning (Figure 2 pipeline with Dev1, Dev2, Dev3 checkpoints)
- Stage 1, cold start: thousands of long chain-of-thought examples in a conversational, first-person style, rewritten by people and checked a second time, used to fine-tune the starting actor
- Stage 2, reasoning RL: R1-Zero's recipe plus a language-consistency reward = target-language words / all words in the chain of thought (Eq. 7); clip ratio 10, temperature 1
- Stage 3, rejection sampling and SFT: about 600k reasoning samples kept only when correct and readable (some judged by DeepSeek-V3 against the ground truth), plus about 200k non-reasoning samples (writing, factual QA, translation, software engineering)
- Stage 4, RL for all scenarios: Reward = Reward_reasoning + Reward_general + Reward_language (Eqs. 8 to 10), temperature 0.7, 1,700 steps, with preference rewards only in the final 400
- Model-based rewards: a helpful reward model trained on 66,000 preference pairs judged on the final summary only, and a point-wise safety reward model trained on 106,000 labelled prompts that sees the whole response
- Reward hacking (Supplementary B.5): with a flawed helpful reward model, reward climbs while real reasoning performance falls
- What each stage bought (Table 3): Dev1 gains instruction following but loses some AIME accuracy; Dev2 restores reasoning; Dev3 lifts general writing; final RL improves AlpacaEval 2.0 by 25% and ArenaHard by 17%
- Training cost as reported (Table 7): about 147K H800 GPU hours, roughly 294K US dollars at 2 dollars per GPU hour, for the RL work on top of an existing base model
- Distillation versus RL, What Failed, and the Limits the Paper Admits
- Distillation: fine-tune open models (Qwen, Llama) on about 800k samples generated by DeepSeek-R1, with SFT only and no RL stage
- Results (Table 15): DeepSeek-R1-Distill-Qwen-1.5B scores 28.9% on AIME 2024 and 83.9% on MATH-500, ahead of GPT-4o and Claude-3.5-Sonnet on these math benchmarks; scores rise with student size
- Distillation versus RL (Table 16): Qwen2.5-32B-Zero trained with large-scale RL reaches QwQ-32B-Preview level, while DeepSeek-R1-Distill-Qwen-32B is far ahead, so distilling a strong teacher beats RL on a smaller model
- Why the base model matters: early RL on 7B dense and 16B MoE models repeated itself and did not improve on AIME; gains appeared at 32B, 230B and 671B
- Unsuccessful attempt 1, process reward models: hard to define a step, hard to label step correctness, and invites reward hacking
- Unsuccessful attempt 2, Monte Carlo Tree Search: the token search space is far larger than in board games and a fine-grained value model is hard to train, so self-search did not keep improving the model
- Capability limits: weak structured output, no tool use, overthinking on easy questions, language mixing for languages other than English and Chinese, sensitivity to prompts, little gain on software engineering
- The big lesson and its catch: pure RL thrives where a reliable verifier exists; for tasks such as writing, where rewards must come from a model, reward hacking makes scaling pure RL an open problem
Open this course in Teachme