DeepSeek-R1-Zero: Rule-Based Rewards, a Minimal Template, and the 'Aha Moment'
DeepSeek-R1: How Pure RL Taught a Model to Reason
- The starting point: DeepSeek-V3-Base, with no supervised fine-tuning before RL, because human-written reasoning patterns might limit what the model explores
- The training template (Table 1): reasoning inside think tags followed by the answer inside answer tags, and deliberately nothing about how to reason
- Accuracy reward: deterministic checks such as an answer in a specified box for math or a compiler and test cases for code
- Format reward: reasoning must sit between the think tags; Reward_rule = Reward_acc + Reward_format with equal weight (Eq. 4)
- Why no neural reward models (outcome- or process-based) for reasoning: they are open to reward hacking at large scale, and retraining them adds compute and complexity
- Results: AIME 2024 pass@1 rose from 15.6% to 77.9%, and 86.7% with self-consistency over 16 samples, beyond the average human competitor
- Self-evolution: average response length climbs steadily to thousands of tokens; reflective words such as 'wait', 'verify', 'mistake' rise 5 to 7 times; 'wait' is almost absent early and spikes after about step 8,000
- The 'aha moment' (Table 2): an intermediate checkpoint stops mid-derivation, flags its own error and re-evaluates step by step
- What R1-Zero got wrong: poor readability, mixing English and Chinese words within one chain of thought, and weak results outside verifiable reasoning such as writing
Straight from base model to RL
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><rect class="tm-surface" x="30" y="40" width="220" height="80" rx="12"/><text class="tm-text" x="140.0" y="74.8" font-size="16" text-anchor="middle">DeepSeek-V3-Base</text><text class="tm-text" x="140.0" y="96.4" font-size="16" text-anchor="middle">pre-trained only</text><line class="tm-line" x1="250" y1="80" x2="300" y2="80" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="305" y="40" width="220" height="80" rx="12"/><text class="tm-text" x="415.0" y="75.125" font-size="15" text-anchor="middle">Supervised fine-tuning</text><text class="tm-text" x="415.0" y="95.375" font-size="15" text-anchor="middle">(usual next step)</text><line class="tm-line-accent" x1="315" y1="50" x2="515" y2="110" stroke-width="3"/><line class="tm-line-accent" x1="315" y1="110" x2="515" y2="50" stroke-width="3"/><line class="tm-line" x1="525" y1="80" x2="575" y2="80" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="580" y="40" width="190" height="80" rx="12"/><text class="tm-text" x="675.0" y="74.8" font-size="16" text-anchor="middle">Reinforcement</text><text class="tm-text" x="675.0" y="96.4" font-size="16" text-anchor="middle">learning</text><rect class="tm-surface" x="30" y="160" width="740" height="120" rx="12"/><text class="tm-text" x="400.0" y="193.2" font-size="17" text-anchor="middle" font-weight="bold">Template (Table 1)</text><text class="tm-text" x="400.0" y="214.79999999999998" font-size="16" text-anchor="middle">User: ‹problem› Assistant:</text><text class="tm-text" x="400.0" y="236.39999999999998" font-size="16" text-anchor="middle"><think> reasoning process </think></text><text class="tm-text" x="400.0" y="258.0" font-size="16" text-anchor="middle"><answer> answer </answer></text><rect class="tm-gold" x="30" y="310" width="740" height="110" rx="12"/><text class="tm-text" x="400.0" y="359.8" font-size="16" text-anchor="middle">Not in the template:</text><text class="tm-text" x="400.0" y="381.40000000000003" font-size="16" text-anchor="middle">'check your work' · 'try another method' · 'think step by step'</text></svg>
```
Rule reward = accuracy + format
| Answer (correct: \(12\)) | Reasoning in think tags | Boxed answer | Accuracy | Format | Total |
| --- | --- | --- | --- | --- | --- |
| Rule (Eq. 4) | format reward | accuracy reward | \(+\) | \(+\) | sum |
| 1 | yes | \(12\) | \(1\) | \(1\) | \(2\) |
| 2 | yes | \(15\) | \(0\) | \(1\) | \(1\) |
| 3 | no | \(12\) | \(1\) | \(0\) | \(1\) |
| Reasoning steps | never read | only the outcome | | | — |
Why not a learned grader?
**Question:** Should a neural reward model score R1-Zero's reasoning?
**Tempting answer:** ~~Yes: a second network that scores answers like a human grader.~~
**Test:** Reward hacking: a policy chasing a grader that likes confident, formula-heavy answers learns to sound confident while solving fewer problems.
**Correct answer:** Use rules: the boxed answer matches the known answer or it does not. Train only on rule-checkable math, code and logic.
**Why:** Rules cannot be flattered, and a learned grader needs costly retraining as the policy changes.
R1-Zero on AIME 2024 during training
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><line class="tm-line" x1="80" y1="40" x2="80" y2="300" stroke-width="2.5"/><line class="tm-line" x1="80" y1="300" x2="480" y2="300" stroke-width="2.5"/><text class="tm-muted" x="280" y="330" font-size="14" text-anchor="middle">training steps</text><text class="tm-muted" x="40" y="170" font-size="14" text-anchor="middle">AIME 2024</text><circle class="tm-accent" cx="90" cy="259" r="6"/><text class="tm-text" x="104" y="285.44" font-size="14" text-anchor="start">start 15.6%</text><polyline class="tm-line" points="90,259 180,222 280,175 380,134 470,97" fill="none" stroke-width="3"/><text class="tm-text" x="478" y="102.45999999999998" font-size="15" text-anchor="start">pass@1 77.9%</text><polyline class="tm-line" points="90,243 180,196 280,144 380,102 470,75" fill="none" stroke-width="3" stroke-dasharray="2 4"/><text class="tm-text" x="478" y="70.57999999999998" font-size="15" text-anchor="start">cons@16 86.7%</text><line class="tm-line" x1="80" y1="196.0" x2="470" y2="196.0" stroke-width="2" stroke-dasharray="6 5"/><text class="tm-muted" x="300" y="216.0" font-size="14" text-anchor="middle">average human competitor</text><line class="tm-line" x1="640" y1="120" x2="640" y2="300" stroke-width="2.5"/><line class="tm-line" x1="640" y1="300" x2="780" y2="300" stroke-width="2.5"/><text class="tm-text" x="710" y="100" font-size="15" text-anchor="middle">answer length</text><polyline class="tm-line-accent" points="645,290 680,265 715,230 750,185 778,140" fill="none" stroke-width="3"/><text class="tm-text-accent" x="710" y="330" font-size="14" text-anchor="middle">rises; never rewarded</text></svg>
```
Reflection emerges, problems remain
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><text class="tm-text" x="400" y="35" font-size="17" text-anchor="middle" font-weight="bold">How often 'wait' appears during training</text><rect class="tm-surface" x="40" y="55" width="230" height="70" rx="12"/><text class="tm-text" x="155.0" y="85.45" font-size="14" text-anchor="middle">steps 0–4,000</text><text class="tm-text" x="155.0" y="104.35000000000001" font-size="14" text-anchor="middle">almost absent</text><rect class="tm-soft" x="285" y="55" width="230" height="70" rx="12"/><text class="tm-text" x="400.0" y="85.45" font-size="14" text-anchor="middle">steps 4,000–7,000</text><text class="tm-text" x="400.0" y="104.35000000000001" font-size="14" text-anchor="middle">now and then</text><rect class="tm-surface" x="530" y="55" width="230" height="70" rx="12"/><text class="tm-text" x="645.0" y="85.45" font-size="14" text-anchor="middle">after step 8,000</text><text class="tm-text" x="645.0" y="104.35000000000001" font-size="14" text-anchor="middle">spikes</text><text class="tm-muted" x="400" y="150" font-size="14" text-anchor="middle">reflective words (wait, mistake, verify, retry, check): ×5–7</text><rect class="tm-surface" x="40" y="175" width="720" height="80" rx="12"/><text class="tm-text" x="400.0" y="210.125" font-size="15" text-anchor="middle">Problem: sum of the real solutions of √(a − √(a + x)) = x</text><text class="tm-text" x="400.0" y="230.375" font-size="15" text-anchor="middle">square both sides → rearrange → a messy fourth-degree equation</text><rect class="tm-surface" x="40" y="265" width="720" height="55" rx="12"/><text class="tm-text" x="400.0" y="297.75" font-size="15" text-anchor="middle">"Wait, wait. Wait. That's an aha moment I can flag here." → re-checks step by step</text><rect class="tm-gold" x="40" y="340" width="225" height="85" rx="12"/><text class="tm-text" x="152.5" y="387.75" font-size="15" text-anchor="middle">Hard to read</text><rect class="tm-gold" x="287" y="340" width="225" height="85" rx="12"/><text class="tm-text" x="399.5" y="377.625" font-size="15" text-anchor="middle">English + Chinese</text><text class="tm-text" x="399.5" y="397.875" font-size="15" text-anchor="middle">mixed in one chain</text><rect class="tm-gold" x="535" y="340" width="225" height="85" rx="12"/><text class="tm-text" x="647.5" y="377.625" font-size="15" text-anchor="middle">Weak at writing,</text><text class="tm-text" x="647.5" y="397.875" font-size="15" text-anchor="middle">open-ended questions</text></svg>
```
Back to course