Distillation versus RL, What Failed, and the Limits the Paper Admits
DeepSeek-R1: How Pure RL Taught a Model to Reason
- Distillation: fine-tune open models (Qwen, Llama) on about 800k samples generated by DeepSeek-R1, with SFT only and no RL stage
- Results (Table 15): DeepSeek-R1-Distill-Qwen-1.5B scores 28.9% on AIME 2024 and 83.9% on MATH-500, ahead of GPT-4o and Claude-3.5-Sonnet on these math benchmarks; scores rise with student size
- Distillation versus RL (Table 16): Qwen2.5-32B-Zero trained with large-scale RL reaches QwQ-32B-Preview level, while DeepSeek-R1-Distill-Qwen-32B is far ahead, so distilling a strong teacher beats RL on a smaller model
- Why the base model matters: early RL on 7B dense and 16B MoE models repeated itself and did not improve on AIME; gains appeared at 32B, 230B and 671B
- Unsuccessful attempt 1, process reward models: hard to define a step, hard to label step correctness, and invites reward hacking
- Unsuccessful attempt 2, Monte Carlo Tree Search: the token search space is far larger than in board games and a fine-grained value model is hard to train, so self-search did not keep improving the model
- Capability limits: weak structured output, no tool use, overthinking on easy questions, language mixing for languages other than English and Chinese, sensitivity to prompts, little gain on software engineering
- The big lesson and its catch: pure RL thrives where a reliable verifier exists; for tasks such as writing, where rewards must come from a model, reward hacking makes scaling pure RL an open problem
Distillation: small students, big teacher
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="30" y="60" width="200" height="90" rx="12"/><text class="tm-text" x="130.0" y="99.8" font-size="16" text-anchor="middle">DeepSeek-R1</text><text class="tm-text" x="130.0" y="121.4" font-size="16" text-anchor="middle">(teacher)</text><line class="tm-line" x1="230" y1="105" x2="280" y2="105" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="285" y="60" width="210" height="90" rx="12"/><text class="tm-text" x="390.0" y="110.6" font-size="16" text-anchor="middle">≈ 800K samples</text><line class="tm-line" x1="495" y1="105" x2="545" y2="105" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="550" y="60" width="220" height="90" rx="12"/><text class="tm-text" x="660.0" y="91.0" font-size="14" text-anchor="middle">Qwen, Llama students</text><text class="tm-text" x="660.0" y="109.9" font-size="14" text-anchor="middle">1.5B to 70B</text><text class="tm-text" x="660.0" y="128.8" font-size="14" text-anchor="middle">SFT only, no RL</text><rect class="tm-surface" x="30" y="190" width="740" height="100" rx="12"/><text class="tm-text" x="400.0" y="225.0" font-size="16" text-anchor="middle" font-weight="bold">Table 15</text><text class="tm-text" x="400.0" y="245.25" font-size="15" text-anchor="middle">DeepSeek-R1-Distill-Qwen-1.5B: AIME 2024 28.9% · MATH-500 83.9%</text><text class="tm-text" x="400.0" y="265.5" font-size="15" text-anchor="middle">ahead of GPT-4o and Claude 3.5 Sonnet on these two math tests</text><rect class="tm-gold" x="30" y="320" width="360" height="100" rx="12"/><text class="tm-text" x="210.0" y="356.0" font-size="14" text-anchor="middle">Caution 1: a math result —</text><text class="tm-text" x="210.0" y="374.9" font-size="14" text-anchor="middle">still trails both on</text><text class="tm-text" x="210.0" y="393.8" font-size="14" text-anchor="middle">GPQA Diamond, LiveCodeBench</text><rect class="tm-gold" x="410" y="320" width="360" height="100" rx="12"/><text class="tm-text" x="590.0" y="365.45" font-size="14" text-anchor="middle">Caution 2: scores rise</text><text class="tm-text" x="590.0" y="384.34999999999997" font-size="14" text-anchor="middle">with student size</text></svg>
```
Distill, or run RL on a small model?
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="40" y="40" width="230" height="80" rx="12"/><text class="tm-text" x="155.0" y="85.6" font-size="16" text-anchor="middle">Qwen2.5-32B-Base</text><line class="tm-line" x1="270" y1="80" x2="320" y2="80" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="325" y="40" width="200" height="80" rx="12"/><text class="tm-text" x="425.0" y="75.125" font-size="15" text-anchor="middle">large-scale RL</text><text class="tm-text" x="425.0" y="95.375" font-size="15" text-anchor="middle">10K+ steps</text><line class="tm-line" x1="525" y1="80" x2="575" y2="80" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="580" y="40" width="190" height="80" rx="12"/><text class="tm-text" x="675.0" y="85.25" font-size="15" text-anchor="middle">Qwen2.5-32B-Zero</text><text class="tm-text" x="400" y="175" font-size="15" text-anchor="middle" font-weight="bold">AIME 2024, same 32B size (Table 16)</text><rect class="tm-surface" x="250" y="190" width="235" height="34" rx="6"/><text class="tm-text" x="240" y="213" font-size="14" text-anchor="end">Qwen2.5-32B-Zero (RL)</text><text class="tm-text" x="493.0" y="213" font-size="14" text-anchor="start">47.0%</text><rect class="tm-surface" x="250" y="234" width="363" height="34" rx="6"/><text class="tm-text" x="240" y="257" font-size="14" text-anchor="end">R1-Distill-Qwen-32B</text><text class="tm-text-accent" x="621.0" y="257" font-size="14" text-anchor="start">72.6%</text><text class="tm-muted" x="400" y="290" font-size="13" text-anchor="middle">QwQ-32B-Preview: roughly level with the RL model</text><rect class="tm-surface" x="40" y="310" width="350" height="60" rx="12"/><text class="tm-text" x="215.0" y="335.45" font-size="14" text-anchor="middle">Small model: distilling</text><text class="tm-text" x="215.0" y="354.34999999999997" font-size="14" text-anchor="middle">beats running RL on it</text><rect class="tm-surface" x="410" y="310" width="350" height="60" rx="12"/><text class="tm-text" x="585.0" y="335.45" font-size="14" text-anchor="middle">Going further: strong base</text><text class="tm-text" x="585.0" y="354.34999999999997" font-size="14" text-anchor="middle">+ large-scale RL</text><rect class="tm-gold" x="40" y="385" width="350" height="55" rx="12"/><text class="tm-text" x="215.0" y="417.4" font-size="14" text-anchor="middle">7B dense, 16B MoE: no AIME gain</text><rect class="tm-green" x="410" y="385" width="350" height="55" rx="12"/><text class="tm-text" x="585.0" y="417.4" font-size="14" text-anchor="middle">32B up to 671B: real gains</text></svg>
```
Two attempts that did not pay off
| Attempt | What went wrong |
| --- | --- |
| PRM: process reward model | scores every reasoning step |
| PRM | hard to define a single step |
| PRM | step labels: automatic too weak, human does not scale |
| PRM | reward hacking + retraining cost |
| PRM | useful for re-ranking top answers only |
| MCTS: tree search | search next steps, guided by a value model |
| MCTS | token space far larger than a board game; caps get stuck |
| MCTS | fine-grained value model too hard to train |
| Fair reading | failed in this setting, not proven impossible |
Limits the paper admits
**1 · Structured output, tools**
Weaker at strict formats; cannot call a search engine or calculator.
**2 · Token efficiency**
Overthinks some simple questions. Longer is not always better.
**3 · Language mixing**
Tuned for English and Chinese; other languages may get English reasoning.
**4 · Prompt sensitivity**
Few-shot prompts hurt. Use zero-shot: state the problem and the output format.
**5 · Software engineering**
Slow evaluations limit RL; little gain over DeepSeek-V3.
Pure RL needs a reliable verifier
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="40" y="30" width="720" height="80" rx="12"/><text class="tm-text" x="400.0" y="65.125" font-size="16" text-anchor="middle" font-weight="bold">Key ingredients</text><text class="tm-text" x="400.0" y="85.375" font-size="15" text-anchor="middle">hard reasoning questions + a reliable way to check answers + enough compute</text><rect class="tm-surface" x="40" y="135" width="350" height="110" rx="12"/><text class="tm-text" x="215.0" y="175.0" font-size="15" text-anchor="middle">Checkable: math, code, logic</text><text class="tm-text" x="215.0" y="195.25" font-size="15" text-anchor="middle">a rule checks the answer</text><text class="tm-text" x="215.0" y="215.5" font-size="15" text-anchor="middle">→ RL works</text><rect class="tm-surface" x="410" y="135" width="350" height="110" rx="12"/><text class="tm-text" x="585.0" y="175.0" font-size="15" text-anchor="middle">Not checkable: writing</text><text class="tm-text" x="585.0" y="195.25" font-size="15" text-anchor="middle">a learned judge scores it</text><text class="tm-text" x="585.0" y="215.5" font-size="15" text-anchor="middle">→ can be gamed (reward hacking)</text><rect class="tm-surface" x="410" y="260" width="350" height="60" rx="12"/><text class="tm-text" x="585.0" y="285.45" font-size="14" text-anchor="middle">R1: human-annotated SFT data,</text><text class="tm-text" x="585.0" y="304.34999999999997" font-size="14" text-anchor="middle">RL for only a few hundred steps</text><text class="tm-text-accent" x="220" y="295" font-size="16" text-anchor="middle">an open challenge</text><rect class="tm-accent" x="40" y="340" width="720" height="95" rx="12"/><text class="tm-text" x="400.0" y="373.5" font-size="14" text-anchor="middle">What checked the answers? rule / tests → likely real gains</text><text class="tm-text" x="400.0" y="392.4" font-size="14" text-anchor="middle">learned judge → look for reward hacking</text><text class="tm-text" x="400.0" y="411.3" font-size="14" text-anchor="middle">small model → learned by RL, or by imitating a teacher?</text></svg>
```
Back to course