The DeepSeek-R1 Pipeline: Cold Start, Two RL Stages, and Rejection-Sampled SFT
DeepSeek-R1: How Pure RL Taught a Model to Reason
- Why a second model: fix R1-Zero's readability and language mixing and extend it beyond verifiable reasoning (Figure 2 pipeline with Dev1, Dev2, Dev3 checkpoints)
- Stage 1, cold start: thousands of long chain-of-thought examples in a conversational, first-person style, rewritten by people and checked a second time, used to fine-tune the starting actor
- Stage 2, reasoning RL: R1-Zero's recipe plus a language-consistency reward = target-language words / all words in the chain of thought (Eq. 7); clip ratio 10, temperature 1
- Stage 3, rejection sampling and SFT: about 600k reasoning samples kept only when correct and readable (some judged by DeepSeek-V3 against the ground truth), plus about 200k non-reasoning samples (writing, factual QA, translation, software engineering)
- Stage 4, RL for all scenarios: Reward = Reward_reasoning + Reward_general + Reward_language (Eqs. 8 to 10), temperature 0.7, 1,700 steps, with preference rewards only in the final 400
- Model-based rewards: a helpful reward model trained on 66,000 preference pairs judged on the final summary only, and a point-wise safety reward model trained on 106,000 labelled prompts that sees the whole response
- Reward hacking (Supplementary B.5): with a flawed helpful reward model, reward climbs while real reasoning performance falls
- What each stage bought (Table 3): Dev1 gains instruction following but loses some AIME accuracy; Dev2 restores reasoning; Dev3 lifts general writing; final RL improves AlpacaEval 2.0 by 25% and ArenaHard by 17%
- Training cost as reported (Table 7): about 147K H800 GPU hours, roughly 294K US dollars at 2 dollars per GPU hour, for the RL work on top of an existing base model
DeepSeek-R1: four stages from the base
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="20" y="160" width="140" height="90" rx="12"/><text class="tm-text" x="90.0" y="200.125" font-size="15" text-anchor="middle">DeepSeek-</text><text class="tm-text" x="90.0" y="220.375" font-size="15" text-anchor="middle">V3-Base</text><line class="tm-line" x1="160" y1="205" x2="182" y2="205" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="185" y="160" width="140" height="90" rx="12"/><text class="tm-muted" x="255" y="210" font-size="15" text-anchor="middle">Box 1</text><rect class="tm-soft" x="335" y="160" width="140" height="90" rx="12"/><text class="tm-muted" x="405" y="210" font-size="15" text-anchor="middle">Box 2</text><line class="tm-line" x1="322" y1="205" x2="333" y2="205" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="485" y="160" width="140" height="90" rx="12"/><text class="tm-muted" x="555" y="210" font-size="15" text-anchor="middle">Box 3</text><line class="tm-line" x1="472" y1="205" x2="483" y2="205" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="635" y="160" width="140" height="90" rx="12"/><text class="tm-muted" x="705" y="210" font-size="15" text-anchor="middle">Box 4</text><line class="tm-line" x1="622" y1="205" x2="633" y2="205" stroke-width="2.5" marker-end="url(#arr)"/><text class="tm-muted" x="400" y="60" font-size="15" text-anchor="middle">R1-Zero left: hard-to-read reasoning · mixed languages · narrow skills</text><rect class="tm-surface" x="185" y="160" width="140" height="90" rx="12"/><text class="tm-text" x="255.0" y="200.775" font-size="13" text-anchor="middle">Cold-start SFT</text><text class="tm-text" x="255.0" y="218.32500000000002" font-size="13" text-anchor="middle">→ Dev1</text><rect class="tm-surface" x="335" y="160" width="140" height="90" rx="12"/><text class="tm-text" x="405.0" y="200.775" font-size="13" text-anchor="middle">Reasoning RL</text><text class="tm-text" x="405.0" y="218.32500000000002" font-size="13" text-anchor="middle">→ Dev2</text><rect class="tm-surface" x="485" y="160" width="140" height="90" rx="12"/><text class="tm-text" x="555.0" y="192.0" font-size="13" text-anchor="middle">Rejection sampling</text><text class="tm-text" x="555.0" y="209.55" font-size="13" text-anchor="middle">+ SFT</text><text class="tm-text" x="555.0" y="227.1" font-size="13" text-anchor="middle">→ Dev3</text><rect class="tm-surface" x="635" y="160" width="140" height="90" rx="12"/><text class="tm-text" x="705.0" y="200.775" font-size="13" text-anchor="middle">RL, all requests</text><text class="tm-text" x="705.0" y="218.32500000000002" font-size="13" text-anchor="middle">→ DeepSeek-R1</text><path class="tm-line-accent" d="M90,250 C90,330 555,330 555,252" fill="none" stroke-width="2.5" marker-end="url(#arr)"/><text class="tm-text-accent" x="320" y="345" font-size="15" text-anchor="middle">box 3 restarts from the base model</text><path class="tm-line" d="M405,160 C405,110 555,110 555,158" fill="none" stroke-width="2" stroke-dasharray="6 5" marker-end="url(#arr)"/><text class="tm-text" x="480" y="105" font-size="14" text-anchor="middle">Dev2: training data only</text><text class="tm-muted" x="400" y="410" font-size="15" text-anchor="middle">Reported RL cost (R1-Zero included): ~147K GPU hours ≈ $294K</text></svg>
```
Box 1: the cold start
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="40" y="30" width="720" height="80" rx="12"/><text class="tm-text" x="400.0" y="55.0" font-size="16" text-anchor="middle" font-weight="bold">Cold-start style</text><text class="tm-text" x="400.0" y="75.25" font-size="15" text-anchor="middle">understand the problem → reason in detail → reflect and check</text><text class="tm-text" x="400.0" y="95.5" font-size="15" text-anchor="middle">first person ('I') · one language · thousands of examples</text><rect class="tm-surface" x="40" y="130" width="220" height="70" rx="12"/><text class="tm-text" x="150.0" y="160.45000000000002" font-size="14" text-anchor="middle">People rewrite</text><text class="tm-text" x="150.0" y="179.35000000000002" font-size="14" text-anchor="middle">raw reasoning</text><line class="tm-line" x1="260" y1="165" x2="290" y2="165" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="290" y="130" width="220" height="70" rx="12"/><text class="tm-text" x="400.0" y="160.45000000000002" font-size="14" text-anchor="middle">A model rewrites</text><text class="tm-text" x="400.0" y="179.35000000000002" font-size="14" text-anchor="middle">many more</text><line class="tm-line" x1="510" y1="165" x2="540" y2="165" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="540" y="130" width="220" height="70" rx="12"/><text class="tm-text" x="650.0" y="160.45000000000002" font-size="14" text-anchor="middle">People check</text><text class="tm-text" x="650.0" y="179.35000000000002" font-size="14" text-anchor="middle">a second time</text><rect class="tm-surface" x="40" y="220" width="330" height="60" rx="12"/><text class="tm-text" x="205.0" y="255.25" font-size="15" text-anchor="middle">DeepSeek-V3-Base + this set → Dev1</text><rect class="tm-surface" x="400" y="220" width="360" height="60" rx="12"/><text class="tm-text" x="580.0" y="245.45000000000002" font-size="14" text-anchor="middle">Purpose: readable for users,</text><text class="tm-text" x="580.0" y="264.35" font-size="14" text-anchor="middle">not teaching reasoning</text><text class="tm-text" x="400" y="315" font-size="15" text-anchor="middle" font-weight="bold">AIME 2024 pass@1</text><rect class="tm-surface" x="160" y="330" width="390" height="34" rx="6"/><text class="tm-text" x="150" y="353" font-size="14" text-anchor="end">R1-Zero</text><text class="tm-text" x="557.5" y="353" font-size="14" text-anchor="start">77.9%</text><rect class="tm-accent" x="160" y="376" width="295" height="34" rx="6"/><text class="tm-text" x="150" y="399" font-size="14" text-anchor="end">Dev1</text><text class="tm-text-accent" x="463.0" y="399" font-size="14" text-anchor="start">59.0% (small set)</text><text class="tm-muted" x="400" y="440" font-size="14" text-anchor="middle">but instruction following: far better</text></svg>
```
Box 2: the language-consistency reward
Reasoning RL on Dev1 with R1-Zero's rules, plus Eq. 7. A question in English; its chain of thought has \(200\) words, \(180\) of them English.
**Eq. 7**
\[ \mathrm{Reward}_{\mathrm{language}} = \frac{\mathrm{Num}(\mathrm{Words}_{\mathrm{target}})}{\mathrm{Num}(\mathrm{Words})} \]
**Substitute**
\[ \mathrm{Reward}_{\mathrm{language}} = \frac{180}{200} \]
**Result**
\[ \frac{180}{200} = 0.9 \]
**All English**
\[ \frac{200}{200} = 1 \]
mixing now costs reward
**Added to the reward**
\[ \mathrm{Reward} = \mathrm{Reward}_{\mathrm{rule}} + \mathrm{Reward}_{\mathrm{language}} \]
**Ablation**
\[ \text{math} \approx \text{same}, \qquad \text{coding: slightly lower} \]
kept for readability
**Dev2**
\[ \text{AIME 2024 pass@1} = 74.0\% \]
most reasoning won back
Box 3: rejection sampling, then SFT
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="20" y="60" width="110" height="60" rx="12"/><text class="tm-text" x="75.0" y="95.6" font-size="16" text-anchor="middle">prompt</text><line class="tm-line" x1="130" y1="90" x2="160" y2="90" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="165" y="60" width="110" height="60" rx="12"/><text class="tm-text" x="220.0" y="95.6" font-size="16" text-anchor="middle">Dev2</text><line class="tm-line" x1="275" y1="90" x2="318" y2="32" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="320" y="20" width="90" height="26" rx="12"/><text class="tm-text" x="365.0" y="37.2" font-size="12" text-anchor="middle">answer 1</text><text class="tm-text-green" x="425" y="38" font-size="16" text-anchor="middle">✓</text><line class="tm-line" x1="275" y1="90" x2="318" y2="62" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="320" y="50" width="90" height="26" rx="12"/><text class="tm-text" x="365.0" y="67.2" font-size="12" text-anchor="middle">answer 2</text><text class="tm-text-accent" x="425" y="68" font-size="16" text-anchor="middle">✗</text><line class="tm-line" x1="275" y1="90" x2="318" y2="92" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="320" y="80" width="90" height="26" rx="12"/><text class="tm-text" x="365.0" y="97.2" font-size="12" text-anchor="middle">answer 3</text><text class="tm-text-green" x="425" y="98" font-size="16" text-anchor="middle">✓</text><line class="tm-line" x1="275" y1="90" x2="318" y2="122" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="320" y="110" width="90" height="26" rx="12"/><text class="tm-text" x="365.0" y="127.2" font-size="12" text-anchor="middle">answer 4</text><text class="tm-text-accent" x="425" y="128" font-size="16" text-anchor="middle">✗</text><line class="tm-line" x1="275" y1="90" x2="318" y2="152" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-soft" x="320" y="140" width="90" height="26" rx="12"/><text class="tm-text" x="365.0" y="157.2" font-size="12" text-anchor="middle">answer 5</text><text class="tm-text-accent" x="425" y="158" font-size="16" text-anchor="middle">✗</text><text class="tm-text" x="560" y="70" font-size="15" text-anchor="middle">keep answers that pass</text><text class="tm-text" x="560" y="92" font-size="15" text-anchor="middle">the check; reject the rest</text><text class="tm-muted" x="560" y="125" font-size="13" text-anchor="middle">(rule, or DeepSeek-V3 as judge)</text><rect class="tm-surface" x="20" y="190" width="370" height="80" rx="12"/><text class="tm-text" x="205.0" y="225.45000000000002" font-size="14" text-anchor="middle">also removed: mixed languages ·</text><text class="tm-text" x="205.0" y="244.35000000000002" font-size="14" text-anchor="middle">long paragraphs · code blocks</text><rect class="tm-surface" x="410" y="190" width="170" height="80" rx="12"/><text class="tm-text" x="495.0" y="224.79999999999998" font-size="16" text-anchor="middle">≈ 600K</text><text class="tm-text" x="495.0" y="246.39999999999998" font-size="16" text-anchor="middle">reasoning</text><rect class="tm-surface" x="600" y="190" width="180" height="80" rx="12"/><text class="tm-text" x="690.0" y="218.0" font-size="12" text-anchor="middle">≈ 200K non-reasoning</text><text class="tm-text" x="690.0" y="234.2" font-size="12" text-anchor="middle">writing · facts ·</text><text class="tm-text" x="690.0" y="250.4" font-size="12" text-anchor="middle">translation · software</text><text class="tm-text" x="500" y="300" font-size="17" text-anchor="middle" font-weight="bold">≈ 800K examples in total</text><rect class="tm-accent" x="20" y="330" width="470" height="90" rx="12"/><text class="tm-text" x="255.0" y="370.125" font-size="15" text-anchor="middle">SFT on DeepSeek-V3-Base (again) → Dev3</text><text class="tm-text" x="255.0" y="390.375" font-size="15" text-anchor="middle">Dev2 was only the data generator</text><rect class="tm-green" x="510" y="330" width="270" height="90" rx="12"/><text class="tm-text" x="645.0" y="361.0" font-size="14" text-anchor="middle">Dev3 vs Dev2:</text><text class="tm-text" x="645.0" y="379.9" font-size="14" text-anchor="middle">AlpacaEval 2.0 ↑</text><text class="tm-text" x="645.0" y="398.8" font-size="14" text-anchor="middle">Aider-Polyglot ↑</text></svg>
```
Box 4: RL for every kind of request
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="40" y="20" width="720" height="60" rx="12"/><text class="tm-text" x="400.0" y="55.6" font-size="16" text-anchor="middle">Reward = Reward_reasoning + Reward_general + Reward_language (Eqs. 8–10)</text><rect class="tm-surface" x="40" y="100" width="350" height="70" rx="12"/><text class="tm-text" x="215.0" y="130.45" font-size="14" text-anchor="middle">reasoning = rule-based</text><text class="tm-text" x="215.0" y="149.35" font-size="14" text-anchor="middle">math · code · logic</text><rect class="tm-surface" x="410" y="100" width="350" height="70" rx="12"/><text class="tm-text" x="585.0" y="130.45" font-size="14" text-anchor="middle">general = reward model + format</text><text class="tm-text" x="585.0" y="149.35" font-size="14" text-anchor="middle">everything else</text><rect class="tm-surface" x="40" y="190" width="350" height="70" rx="12"/><text class="tm-text" x="215.0" y="220.45000000000002" font-size="14" text-anchor="middle">Helpful RM: preference pairs</text><text class="tm-text" x="215.0" y="239.35000000000002" font-size="14" text-anchor="middle">reads the final summary only</text><rect class="tm-surface" x="410" y="190" width="350" height="70" rx="12"/><text class="tm-text" x="585.0" y="220.45000000000002" font-size="14" text-anchor="middle">Safety RM: safe / unsafe labels</text><text class="tm-text" x="585.0" y="239.35000000000002" font-size="14" text-anchor="middle">reads the whole response</text><rect class="tm-surface" x="60" y="300" width="680" height="36" rx="6"/><rect class="tm-surface" x="580" y="300" width="160" height="36" rx="6"/><text class="tm-text" x="60" y="290" font-size="14" text-anchor="start">1,700 steps at temperature 0.7</text><text class="tm-text-accent" x="660.0" y="290" font-size="13" text-anchor="middle">last 400: preference rewards on</text><text class="tm-muted" x="400" y="360" font-size="14" text-anchor="middle">longer use of the learned judge → reward hacking</text><rect class="tm-green" x="140" y="375" width="520" height="60" rx="12"/><text class="tm-text" x="400.0" y="400.45" font-size="14" text-anchor="middle">Dev3 → R1: AlpacaEval 2.0 +25 points · ArenaHard +17</text><text class="tm-text" x="400.0" y="419.34999999999997" font-size="14" text-anchor="middle">math and code: only slight change</text></svg>
```
Back to course