Why DeepSeek-R1 Matters: The Question, the Two Models, and the Roadmap
DeepSeek-R1: How Pure RL Taught a Model to Reason
- The question the paper asks: can a language model get better at hard reasoning if we only tell it whether its final answer was right, without ever showing it how to think?
- Why that question matters: worked solutions written by people are slow and expensive to collect, and they may cap a model at the way people happen to solve problems
- The two models in the paper: a first experiment trained only by being rewarded for right answers, and a polished model built on top of what that experiment taught
- What surprised the authors: the experimental model started writing longer answers and checking its own work without being told to
- How the rest of the paper turns that discovery into a readable, general-purpose assistant and then into small models anyone can run
- Roadmap: the learning algorithm, the first experiment and what emerged, the full training pipeline, and finally shrinking the model and the limits the authors admit
Two ways to coach a solver
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="40" y="70" width="330" height="190" rx="12"/><text class="tm-text" x="205.0" y="138.2" font-size="17" text-anchor="middle" font-weight="bold">Way 1</text><text class="tm-text" x="205.0" y="159.79999999999998" font-size="16" text-anchor="middle">Binder of worked solutions</text><text class="tm-text" x="205.0" y="181.39999999999998" font-size="16" text-anchor="middle">written by experts</text><text class="tm-text" x="205.0" y="203.0" font-size="16" text-anchor="middle">→ study every step</text><rect class="tm-surface" x="430" y="70" width="330" height="190" rx="12"/><text class="tm-text" x="595.0" y="138.2" font-size="17" text-anchor="middle" font-weight="bold">Way 2</text><text class="tm-text" x="595.0" y="159.79999999999998" font-size="16" text-anchor="middle">Problems + an answer key</text><text class="tm-text" x="595.0" y="181.39999999999998" font-size="16" text-anchor="middle">feedback: right or wrong</text><text class="tm-text" x="595.0" y="203.0" font-size="16" text-anchor="middle">no worked solutions</text><text class="tm-text-accent" x="595" y="290" font-size="16" text-anchor="middle">The paper's question: can a model learn to reason this way?</text><rect class="tm-gold" x="140" y="330" width="520" height="80" rx="12"/><text class="tm-text" x="400.0" y="375.95" font-size="17" text-anchor="middle">Same network design · new: training after pre-training</text></svg>
```
Why try right-or-wrong feedback?
**Reason 1: cost**
Worked solutions need an expert's time, many times over. An answer key is a single number, or a set of tests the code must pass.
**Reason 2: a hidden ceiling**
Copying human solutions caps the model at how people happened to solve problems. They also omit double-checking and backtracking.
**The bet**
Hard problems + a reliable check + many tries → reward what works. Any reasoning habit is the model's own discovery.
Two models, not one
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="40" y="40" width="330" height="170" rx="12"/><text class="tm-text" x="205.0" y="87.39999999999999" font-size="17" text-anchor="middle" font-weight="bold">DeepSeek-R1-Zero</text><text class="tm-text" x="205.0" y="109.0" font-size="16" text-anchor="middle">Pure experiment</text><text class="tm-text" x="205.0" y="130.6" font-size="16" text-anchor="middle">from DeepSeek-V3-Base</text><text class="tm-text" x="205.0" y="152.2" font-size="16" text-anchor="middle">only right/wrong feedback</text><text class="tm-text" x="205.0" y="173.8" font-size="16" text-anchor="middle">math · coding · logic</text><line class="tm-line" x1="375" y1="125" x2="425" y2="125" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="430" y="40" width="330" height="170" rx="12"/><text class="tm-text" x="595.0" y="98.19999999999999" font-size="17" text-anchor="middle" font-weight="bold">DeepSeek-R1</text><text class="tm-text" x="595.0" y="119.79999999999998" font-size="16" text-anchor="middle">Polished assistant</text><text class="tm-text" x="595.0" y="141.39999999999998" font-size="16" text-anchor="middle">several training stages</text><text class="tm-text" x="595.0" y="163.0" font-size="16" text-anchor="middle">readable, general-purpose</text><rect class="tm-surface" x="40" y="240" width="330" height="90" rx="12"/><text class="tm-text" x="205.0" y="280.125" font-size="15" text-anchor="middle">Emerged unasked: longer answers,</text><text class="tm-text" x="205.0" y="300.375" font-size="15" text-anchor="middle">self-checks, an 'aha moment'</text><rect class="tm-gold" x="40" y="345" width="330" height="80" rx="12"/><text class="tm-text" x="205.0" y="380.125" font-size="15" text-anchor="middle">Rough edges: hard to read,</text><text class="tm-text" x="205.0" y="400.375" font-size="15" text-anchor="middle">English and Chinese mixed</text><line class="tm-line-accent" x1="375" y1="385" x2="520" y2="215" stroke-width="2.5" stroke-dasharray="6 5" marker-end="url(#arr)"/></svg>
```
The road ahead: four stops
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="30" y="150" width="168" height="150" rx="12"/><text class="tm-text" x="114.0" y="210.0" font-size="16" text-anchor="middle" font-weight="bold">Stop 1</text><text class="tm-text" x="114.0" y="230.25" font-size="15" text-anchor="middle">Learning algorithm</text><text class="tm-text" x="114.0" y="250.5" font-size="15" text-anchor="middle">GRPO</text><rect class="tm-surface" x="222" y="150" width="168" height="150" rx="12"/><text class="tm-text" x="306.0" y="210.0" font-size="16" text-anchor="middle" font-weight="bold">Stop 2</text><text class="tm-text" x="306.0" y="230.25" font-size="15" text-anchor="middle">First experiment</text><text class="tm-text" x="306.0" y="250.5" font-size="15" text-anchor="middle">R1-Zero</text><line class="tm-line" x1="200" y1="225" x2="220" y2="225" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="414" y="150" width="168" height="150" rx="12"/><text class="tm-text" x="498.0" y="210.0" font-size="16" text-anchor="middle" font-weight="bold">Stop 3</text><text class="tm-text" x="498.0" y="230.25" font-size="15" text-anchor="middle">Full pipeline</text><text class="tm-text" x="498.0" y="250.5" font-size="15" text-anchor="middle">DeepSeek-R1</text><line class="tm-line" x1="392" y1="225" x2="412" y2="225" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-accent" x="606" y="150" width="168" height="150" rx="12"/><text class="tm-text" x="690.0" y="210.0" font-size="16" text-anchor="middle" font-weight="bold">Stop 4</text><text class="tm-text" x="690.0" y="230.25" font-size="15" text-anchor="middle">Shrink + limits</text><text class="tm-text" x="690.0" y="250.5" font-size="15" text-anchor="middle">small models</text><line class="tm-line" x1="584" y1="225" x2="604" y2="225" stroke-width="2.5" marker-end="url(#arr)"/></svg>
```
Back to course