GRPO: Group Relative Policy Optimization and Why It Drops PPO's Value Model
DeepSeek-R1: How Pure RL Taught a Model to Reason
- Vocabulary for an LLM as an RL agent: the policy is the model, an action is the generated output, the reward scores the output, and the advantage says how much better than expected an output was
- The GRPO recipe: for each question q, sample a group of G outputs from the old policy, score each one, and compute A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G) (paper Eq. 3)
- The GRPO objective (Eq. 1) read term by term: the probability ratio pi_theta(o_i|q) / pi_theta_old(o_i|q), the min with a clipped ratio in [1 - epsilon, 1 + epsilon], averaging over the group, and minus beta times the KL divergence to a reference policy
- The KL estimator (Eq. 2): ratio pi_ref / pi_theta minus log of that ratio minus 1, which is always non-negative, and why the reference policy is refreshed to the latest policy every 400 steps
- PPO for contrast: a value model of similar size to the policy plus Generalized Advantage Estimation (GAE); its memory and compute overhead; why predicting the final reward from a partial chain of thought is especially hard when later text revises earlier text
- Why a per-token KL reward (as in PPO-based RLHF) implicitly penalizes long responses, while GRPO's loss-level KL does not block response length from growing
- Evidence in the paper (Supplementary A.3): PPO with the default GAE lambda of 0.95 trails GRPO on MATH; tuned to lambda = 1.0 it nearly matches, at extra tuning cost
- DeepSeek-R1-Zero's RL settings: learning rate 3e-6, KL coefficient 0.001, temperature 1, 16 outputs per question, 32 questions per step (batch 512), max length 32,768 tokens raised to 65,536 after step 8.2k, 10,400 steps in total
Four words for RL on a model
**Policy \(\pi_\theta\)**
The model itself, choosing what to write; \(\theta\) = its adjustable weights.
**Action**
One complete output: everything written in answer to one question.
**Reward**
A score for that output; for now \(1\) if right, \(0\) if wrong.
**Advantage**
How much better or worse than expected this output did. Positive → more likely; negative → less likely.
Group-relative advantage, worked
One question \(q\); a group of \(G = 4\) outputs with rewards \(r = (1, 0, 0, 1)\).
**Sample and score**
\[ q \;\to\; \{o_1,\dots,o_G\} \;\to\; \{r_1,\dots,r_G\} \]
**Advantage (Eq. 3)**
\[ A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)} \]
**Mean**
\[ \operatorname{mean} = \frac{1+0+0+1}{4} = \frac{1}{2} \]
**Standard deviation**
\[ \operatorname{std} = \sqrt{\tfrac{1}{4}\textstyle\sum_{i}\big(r_i - \tfrac12\big)^2} = \sqrt{\tfrac14} = \tfrac12 \]
every reward is 1/2 from the mean
**Output 1**
\[ A_1 = \frac{1 - \frac12}{\frac12} = +1 \]
**Output 2**
\[ A_2 = \frac{0 - \frac12}{\frac12} = -1 \]
**Whole group**
\[ (A_1, A_2, A_3, A_4) = (+1,\,-1,\,-1,\,+1) \]
1 and 4 more likely; 2 and 3 less likely
**Trap: all equal**
\[ r = (1,1,1,1) \;\Rightarrow\; r_i - \operatorname{mean} = 0 \;\Rightarrow\; A_i = 0 \]
no learning signal this round
Reading the GRPO objective
The GRPO objective (Eq. 1), built piece by piece; training makes it as large as possible.
**Piece 1: ratio**
\[ \frac{\pi_\theta(o_i \mid q)}{\pi_{\theta_{\mathrm{old}}}(o_i \mid q)} \]
above 1 = the update made \(o_i\) more likely
**Piece 2: times advantage**
\[ \frac{\pi_\theta(o_i \mid q)}{\pi_{\theta_{\mathrm{old}}}(o_i \mid q)}\; A_i \]
**Piece 3: clip, take the smaller**
\[ \min\!\Big(\tfrac{\pi_\theta}{\pi_{\theta_{\mathrm{old}}}} A_i,\; \mathrm{clip}\big(\tfrac{\pi_\theta}{\pi_{\theta_{\mathrm{old}}}},\, 1-\varepsilon,\, 1+\varepsilon\big) A_i\Big),\quad \varepsilon = 0.2 \]
**Worked: \(A_1 = +1\), ratio \(1.5\)**
\[ \min(1.5 \times 1,\; 1.2 \times 1) = 1.2 \]
no extra credit beyond 1.2 in this update
**Piece 4: average over the group**
\[ \frac{1}{G}\sum_{i=1}^{G} \min\!\Big(\tfrac{\pi_\theta}{\pi_{\theta_{\mathrm{old}}}} A_i,\; \mathrm{clip}\big(\tfrac{\pi_\theta}{\pi_{\theta_{\mathrm{old}}}},\, 1-\varepsilon,\, 1+\varepsilon\big) A_i\Big) \]
**Piece 5: minus the KL penalty**
\[ \mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\Big[\frac{1}{G}\sum_{i=1}^{G}\Big(\min(\cdots) - \beta\, \mathbb{D}_{\mathrm{KL}}\big(\pi_\theta \,\|\, \pi_{\mathrm{ref}}\big)\Big)\Big] \]
The KL leash, estimated
Penalty for drifting from a frozen reference model \(\pi_{\mathrm{ref}}\) (Eq. 2).
**Define r**
\[ r = \frac{\pi_{\mathrm{ref}}(o_i \mid q)}{\pi_\theta(o_i \mid q)} \]
**Estimate**
\[ \mathbb{D}_{\mathrm{KL}}\big(\pi_\theta \,\|\, \pi_{\mathrm{ref}}\big) \approx r - \ln r - 1 \]
**Models agree: \(r = 1\)**
\[ 1 - \ln 1 - 1 = 1 - 0 - 1 = 0 \]
no penalty
**Reference twice as likely: \(r = 2\)**
\[ 2 - \ln 2 - 1 \approx 2 - 0.69 - 1 \approx 0.31 \]
never negative; grows with disagreement
**R1-Zero settings**
\[ \beta = 0.001, \qquad \pi_{\mathrm{ref}} \leftarrow \pi_\theta \ \text{every } 400 \text{ steps} \]
**Placement**
\[ \text{GRPO: } -\beta\,\mathbb{D}_{\mathrm{KL}} \text{ in the objective, not a per-token cost in the reward} \]
PPO's value model vs GRPO's group
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M0,0 L10,5 L0,10 z"/></marker></defs><text class="tm-text" x="200" y="30" font-size="20" text-anchor="middle" font-weight="bold">PPO</text><rect class="tm-surface" x="30" y="50" width="140" height="60" rx="12"/><text class="tm-text" x="100.0" y="85.6" font-size="16" text-anchor="middle">Policy</text><line class="tm-line" x1="170" y1="80" x2="220" y2="80" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="225" y="50" width="150" height="60" rx="12"/><text class="tm-text" x="300.0" y="75.45" font-size="14" text-anchor="middle">Value model</text><text class="tm-text" x="300.0" y="94.35000000000001" font-size="14" text-anchor="middle">(~ policy size)</text><line class="tm-line" x1="300" y1="110" x2="300" y2="150" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="225" y="150" width="150" height="50" rx="12"/><text class="tm-text" x="300.0" y="179.9" font-size="14" text-anchor="middle">GAE → advantage</text><rect class="tm-surface" x="30" y="150" width="180" height="110" rx="12"/><text class="tm-text" x="120.0" y="183.22500000000002" font-size="13" text-anchor="middle">Hard: predict the final</text><text class="tm-text" x="120.0" y="200.77500000000003" font-size="13" text-anchor="middle">reward from a partial</text><text class="tm-text" x="120.0" y="218.32500000000002" font-size="13" text-anchor="middle">answer that may still</text><text class="tm-text" x="120.0" y="235.87500000000003" font-size="13" text-anchor="middle">change course</text><text class="tm-text" x="600" y="30" font-size="20" text-anchor="middle" font-weight="bold">GRPO</text><rect class="tm-surface" x="430" y="50" width="140" height="60" rx="12"/><text class="tm-text" x="500.0" y="85.6" font-size="16" text-anchor="middle">Policy</text><line class="tm-line" x1="570" y1="80" x2="610" y2="80" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="615" y="50" width="160" height="60" rx="12"/><text class="tm-text" x="695.0" y="75.45" font-size="14" text-anchor="middle">G outputs</text><text class="tm-text" x="695.0" y="94.35000000000001" font-size="14" text-anchor="middle">(group)</text><line class="tm-line" x1="695" y1="110" x2="695" y2="150" stroke-width="2.5" marker-end="url(#arr)"/><rect class="tm-surface" x="600" y="150" width="180" height="50" rx="12"/><text class="tm-text" x="690.0" y="179.9" font-size="14" text-anchor="middle">mean, std → advantage</text><text class="tm-text-green" x="600" y="230" font-size="15" text-anchor="middle">no value model</text><rect class="tm-surface" x="30" y="280" width="345" height="60" rx="12"/><text class="tm-text" x="202.5" y="305.45" font-size="14" text-anchor="middle">KL: per-token cost in the reward</text><text class="tm-text" x="202.5" y="324.34999999999997" font-size="14" text-anchor="middle">→ longer answers pay more</text><rect class="tm-surface" x="430" y="280" width="345" height="60" rx="12"/><text class="tm-text" x="602.5" y="305.45" font-size="14" text-anchor="middle">KL: in the objective</text><text class="tm-text" x="602.5" y="324.34999999999997" font-size="14" text-anchor="middle">→ length free to grow</text><rect class="tm-accent" x="30" y="360" width="745" height="75" rx="12"/><text class="tm-text" x="402.5" y="392.625" font-size="15" text-anchor="middle">Supplementary A.3, MATH task: PPO with λ = 0.95 trails GRPO;</text><text class="tm-text" x="402.5" y="412.875" font-size="15" text-anchor="middle">PPO tuned to λ = 1.0 comes close, after extra tuning, with the second network</text></svg>
```
Back to course