RL with Verifiable Rewards: GRPO and Reasoning Models#

The final act of the deletion narrative. DPO deleted the sampling loop and paid for it in staleness; this era brings the loop back — and instead deletes the two learned auxiliaries. The reward model becomes a program, and the critic becomes a group statistic. What remains is nearly the oldest algorithm in Section 3: REINFORCE with a baseline, pointed at math problems. This recipe — RL with verifiable rewards (RLVR; Lambert et al. 2024) trained with GRPO (Shao et al. 2024) — is what produced the reasoning models of 2024–25.

From reward model to verifier#

Swap $r_\phi(x, y)$, a learned proxy, for $R(x, y)$, a program: exact-match on the boxed answer of a math problem, a unit-test suite for generated code, a proof checker, a compiler. What changes and what doesn’t:

• Unchanged: the reward is still terminal and sparse (Section 4.1’s MDP is untouched — if anything the reward got sparser, typically binary); the objective is still expected reward, usually still with a KL leash to a reference.
• Changed: the reward can no longer be flattered. A verifier does not care about confident tone or bullet points, so the stylistic reward hacking of Section 4.2 dies instantly. (Hacking itself does not die — it relocates; second deep dive.)
• Changed, and decisive: reward queries are now free. The entire economics of 4.2 — reward model as a machine for stretching $10^5$ human judgments into $10^7$ reward queries — was a workaround for expensive evaluation. When grading costs nothing, you grade fresh samples from the current policy at every step, and the offline compromise that defined DPO becomes pointless. On-policy RL is back because it can be.

GRPO: the critic becomes a group statistic#

The expensive survivor in PPO-RLHF was the critic: a full transformer trained to predict values of partial sequences from a terminal, sparse signal — the hardest regression in the pipeline, existing only to provide a baseline for variance reduction. GRPO’s observation: with free reward queries, a baseline can be had by brute force. Sample a group of $G$ responses $o_1, \dots, o_G$ to the same prompt, grade them all, and let the group tell you what “typical” looks like:

with the same $\hat{A}_i$ assigned to every token of response $i$. The policy update is then PPO’s clipped surrogate with per-token ratios $\rho_{i,t} = \pi_\theta(o_{i,t} \mid \cdot)\,/\,\pi_{\theta_{\text{old}}}(o_{i,t} \mid \cdot)$:

where $\hat{\mathbb{D}}_{i,t}$ is a per-token KL estimate to the frozen reference — the nonnegative $k_3$ estimator, added to the loss rather than folded into the reward. Both choices are ones the two-trust-regions deep dive flagged as “implementations differ, and it matters”: GRPO picks $k_3$ and loss-placement, which also means there is no shaped per-token reward left at all — with no critic to feed, the KL-in-reward channel lost its purpose. The full formalization, with the estimator’s properties, is in the mathematics companion; this page stays at the level of what the algorithm does and where it creaks.

Three connections that keep this from looking like magic:

1. It is REINFORCE with a baseline. The group mean is the classical prompt-dependent baseline $b(x)$ from Section 3, estimated by Monte Carlo with $G$ samples instead of a learned network. The closely related RLOO (Ahmadian et al. 2024) uses the leave-one-out mean — excluding $R_i$ from its own baseline — which restores exact unbiasedness; GRPO’s version accepts a small bias for simplicity.
2. It lives in the bandit reading. One reward per response, broadcast uniformly to all its tokens, no bootstrapping — exactly the regime where the token-vs-sequence deep dive showed the sequence-level view is exact. GRPO makes no attempt at per-token credit assignment; it bets that with enough cheap samples, you don’t need any.
3. The memory bill. Policy (training), reference (frozen), and $\pi_{\theta_{\text{old}}}$ is just cached log-probs from generation time. Two networks, one training — compare four in Section 4.2.

The code#

The advantage computation is genuinely this small, and the verified run below demonstrates two of its real behaviors — one benign, one that became a paper:

import torch
torch.manual_seed(0)

def grpo_advantages(rewards, eps=1e-4, normalize_std=True):
    """rewards: (G,) verifier scores for the G responses to ONE prompt."""
    adv = rewards - rewards.mean()          # group mean as baseline
    if normalize_std:
        adv = adv / (rewards.std() + eps)   # the knob Dr. GRPO removes
    return adv

# A group of G = 8 responses to one math problem, graded 0/1 by the verifier
rewards = torch.tensor([1., 0., 0., 1., 0., 0., 0., 1.])   # 3 of 8 correct
adv = grpo_advantages(rewards)
print("advantages:        ", adv.round(decimals=3))

# Every token of response i carries adv[i]; GRPO then averages within the
# response, so the per-token weight is adv[i] / |o_i|. Watch what length does:
lengths = torch.tensor([120., 340., 95., 210., 55., 400., 130., 88.])
per_token = adv / lengths
print(f"per-token weight of wrong answer, len  55: {per_token[4]:.4f}")
print(f"per-token weight of wrong answer, len 400: {per_token[5]:.4f}")

# Degenerate groups: all correct (or all wrong) means zero advantage everywhere
print("all-correct group: ", grpo_advantages(torch.ones(8)).round(decimals=3))

advantages:         tensor([ 1.2070, -0.7240, -0.7240,  1.2070, -0.7240, -0.7240, -0.7240,  1.2070])
per-token weight of wrong answer, len  55: -0.0132
per-token weight of wrong answer, len 400: -0.0018
all-correct group:  tensor([0., 0., 0., 0., 0., 0., 0., 0.])

Read the three prints in order. First: correct responses share one positive advantage, incorrect ones share one negative advantage — group-relative, exactly as designed. Second: because of the $1/|o_i|$ normalization, the wrong answer of length 55 is punished about seven times harder per token than the wrong answer of length 400. Being long dilutes punishment — an incentive for incorrect responses to ramble, measured in two numbers. This is the masking/normalization decision the Section 4.1 primitive promised would return; the first deep dive gives it the full treatment. Third: a group that is all-correct (or all-wrong) has zero advantage everywhere — no gradient. Prompts the policy has mastered, or has no hope on, contribute nothing; effective batch size is set by the prompts of intermediate difficulty, which is why practical pipelines filter for them and why training has a curriculum flavor even when nobody designed one (DAPO’s “dynamic sampling” is this observation turned into a method).

What “reasoning” means here#

Both this page’s title and its case study use words the field rarely defines. Before the case study, definitions — operational ones, because that is all the field has.

A reasoning task is one whose answer cannot be produced by retrieval or single-step association, but requires composing intermediate results: multi-step mathematics, code with dependencies, deduction, planning. A reasoning model — the term as used since late 2024 — is a model trained, by exactly this page’s recipe, to spontaneously produce long chains of thought with reflection, self-verification, and backtracking before committing to an answer. Its defining behavior is adaptive test-time compute: short traces for easy questions, thousands of deliberation tokens for hard ones, where earlier models spent roughly constant compute regardless of difficulty.

The mechanism under both terms is the chain of thought (CoT): the intermediate tokens emitted between reading the problem and stating the answer. Historically it entered as a prompting trick — worked examples in the prompt (Wei et al. 2022), or merely appending “let’s think step by step” (Kojima et al. 2022) — and became a trained behavior, first through SFT on solution traces, then through the RLVR of this page. Same phenomenon, three ways of eliciting it, and three complementary explanations of why it works:

1. The computational view. A transformer’s forward pass is a fixed computation: the same layers, the same serial depth, whether the question is trivial or hard. Emitting a token changes that — each generated token triggers a fresh forward pass conditioned on everything written so far, so the context window becomes a scratchpad: external working memory that the fixed-depth circuit writes to and reads back. CoT converts sequence length into serial computation depth, and this is a theorem rather than a metaphor: fixed-depth transformers provably cannot solve certain inherently serial problems in one pass, yet with polynomially many CoT steps can simulate any polynomial-time computation (Merrill & Sabharwal 2023; Feng et al. 2023 for the arithmetic and dynamic-programming version). A model that “thinks longer” is literally running a longer program.

2. The probabilistic view. Treat the trace $z$ as a latent variable between question $x$ and answer $a$:

Direct answering demands the marginal in one shot; CoT samples a path $z$ and answers conditionally, and each conditional factor is simple even when the marginal is not. This view buys self-consistency for free (Wang et al. 2023): sample several traces, majority-vote the answers — a crude Monte-Carlo estimate of the marginal, reliably better than any single path. It also names the two axes of test-time compute: sequential (a longer $z$) and parallel (more samples of $z$, aggregated by voting or a verifier).

3. The RL view — this chapter’s. In the Section 4.1 MDP, CoT tokens are actions that receive no direct reward: the verifier grades only the final answer, and the trace is a trajectory through state space en route to it. RLVR therefore does not teach reasoning steps individually; it reweights whole trajectory strategies — decomposition, re-checking, backtracking — by their correlation with terminal success. Read through the other two views, the case study below loses its mystery: length growth is the policy discovering that longer programs (view 1) and better-explored paths (view 2) reach reward more often.

Bracketing the philosophy. Whether any of this is “real” reasoning is contested. The skeptical reading — pattern-matching over reasoning-shaped text — has evidence: performance degrades under distribution shifts (irrelevant clauses, unusual number ranges) in ways human reasoning would shrug off. The pragmatic reading, which the field has adopted, defines reasoning behaviorally and notes that the computational view’s theorems show something does change when models generate intermediate steps. For this lecture: “reasoning” names a capability regime, CoT is its mechanism, and the RL on this page is what unlocked it at scale.

Case study: DeepSeek-R1#

The result that made this recipe famous (DeepSeek-AI 2025). R1-Zero: take a base model — no SFT stage at all — and run GRPO with two rule-based rewards (answer correctness, plus a format reward for keeping reasoning inside designated tags). Over training, responses grow from hundreds to thousands of tokens without any length being asked for; the model develops re-checking, backtracking, and trying alternative approaches — including the much-quoted “aha moment” where it interrupts its own solution to flag a mistake. Benchmark movement was dramatic (AIME pass@1 from ~16% to ~78% over the run). R1 proper wraps this in a multi-stage pipeline — a small “cold start” SFT on long reasoning traces first, then reasoning RL, then a general RLHF-style stage — mostly to fix R1-Zero’s readability problems (language mixing, formatless rambling).

Three glosses worth mentioning:

• What “emergence” means here, mechanically. Nothing appeared from nowhere: reflection, verification phrases, and long derivations all exist in the base model’s distribution at low probability (they are all over pretraining data). RLVR raises the probability of behaviors that correlate with getting the answer right. That is remarkable — the reward never mentions reflection — but it is probability reweighting of existing capabilities, not creation ex nihilo, and follow-up work (including the Dr. GRPO authors) found “aha”-style phrasing already present in base-model samples before any RL.
• Longer is not the same as better. Length growth is partly the useful thing (harder problems need more steps) and partly the artifact quantified in the code block above. The two are entangled in every training curve; treat “response length went up” as evidence of neither success nor failure by itself.
• The distillation coda. Small models trained by supervised distillation on R1’s traces beat small models trained by RL directly — for compact models, imitating a strong reasoner outperformed exploring with a verifier. A usefully deflationary note for anyone planning to apply RLVR at modest scale: sometimes the best use of RL is to run it once, at size, and distill.

Handoff#

Take stock of what made this page’s recipe work: transitions were deterministic (Section 4.1), episodes were single-turn and cheaply resettable, and grading was free and instantaneous. Single-turn verifiable generation is RL on easy mode. Agents — models that browse, call tools, and act over many turns — surrender every one of those luxuries at once, and the problems this chapter kept dodging (stochastic dynamics, credit assignment over genuinely sequential decisions, sparse reward over long horizons) return in force. Section 4.5: agents and multi-turn RL.

References.

• Shao et al. (2024), DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO);
• DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning;
• Lambert et al. (2024), Tülu 3: Pushing Frontiers in Open Language Model Post-Training (RLVR);
• Liu et al. (2025), Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO);
• Ahmadian et al. (2024), Back to Basics: Revisiting REINFORCE-Style Optimization for RLHF (RLOO);
• Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale;
• Kool et al. (2019), Buy 4 REINFORCE Samples, Get a Baseline for Free!
• Wei et al. (2022), *Chain-of-Thought Prompting Elicits Reasoning in Large Language
• Models*; Kojima et al. (2022), Large Language Models are Zero-Shot Reasoners;
• Wang et al. (2023), Self-Consistency Improves Chain of Thought Reasoning;
• Merrill & Sabharwal (2023), The Expressive Power of Transformers with Chain of Thought;
• Feng et al. (2023), Towards Revealing the Mystery behind Chain of Thought;
• Turpin et al. (2023), Language Models Don’t Always Say What They Think.