RL for LLMs, Reasoning, and Agents#

(Lecture time: 0:50–1:10 — the section this school is about)

The mapping#

An LLM is a policy:

RL concept	LLM instantiation
State $s$	Prompt + tokens generated so far
Action $a$	Next token (or full response)
Policy $\pi_\theta$	The language model
Reward $r$	Human preference score, verifier output, task success
Episode	One complete response (or multi-turn interaction)

RLHF: RL from human feedback#

The pipeline that turned base models into assistants (InstructGPT, 2022):

The RLHF pipeline

Why a reward model instead of asking humans directly? RL needs millions of reward queries; humans are slow and expensive. So: collect comparisons (“response A > B”), fit a reward model $r_\phi$ (Bradley–Terry loss on preference pairs), then optimize against it with PPO — with a KL penalty to the SFT model to prevent the policy from wandering into degenerate text that games the reward model.

Deep dive: the RLHF objective

$$ \max_\theta\ \mathbb{E}_{x \sim D,\, y \sim \pi_\theta} \big[ r_\phi(x, y) \big] - \beta\, \mathbb{D}_{\mathrm{KL}}\!\big( \pi_\theta(\cdot \mid x)\ \|\ \pi_{\text{ref}}(\cdot \mid x) \big) $$ The KL term is doing real work: reward models are imperfect proxies, and unconstrained optimization against a proxy invites reward hacking (Goodhart's law, an old friend to every experimentalist).

DPO: skip the RL loop#

Direct Preference Optimization (2023) observes that the KL-regularized RLHF objective has a closed-form optimal policy, which lets you rewrite the whole thing as a simple classification-style loss directly on preference pairs — no reward model, no PPO, no sampling during training. Widely used because it’s stable and cheap; the trade-off is less flexibility (purely offline, tied to the preference dataset).

RLVR: verifiable rewards → reasoning models#

The 2024–2025 shift: for math and code, correctness is checkable. Unit tests pass or fail; the final answer matches or it doesn’t. That gives a reward signal that is ungameable in a way learned reward models are not — so you can push optimization much harder.

Recipe (DeepSeek-R1 style):

Sample many chain-of-thought attempts per problem.
Reward = verifier output (correct / incorrect), plus format rewards.
Policy-gradient update — GRPO: instead of a learned critic, use the group of samples for the same prompt as its own baseline (advantage = your score minus the group mean). Cheaper than PPO (no value network) and well-suited to LLMs.

Emergent result: models spontaneously learn to generate long deliberation, check their own work, and backtrack — “reasoning” as an outcome of RL, not an architecture change.

**Why this matters for science:** wherever you have a cheap, reliable verifier — symbolic math, simulators, property checkers, conservation laws — you have the raw material for RLVR-style training on *your* domain.

Agents: the open frontier#

Agentic systems act over long horizons: call tools, run code, browse, iterate. RL-wise this is the hard regime —

Credit assignment over hundreds of steps: which tool call caused the success?
Sparse rewards: often only terminal task success/failure.
Environment cost: each rollout may take minutes and real resources.
Reward specification: “did the agent do a good job on this research task?” is itself hard to verify.

Current practice mixes process rewards, learned verifiers, and RLVR where possible. Expect the other lectures this week to pick up exactly here.