Why RL, Why Now#
(Lecture time: 0:00–0:10)
The punchline first#
The “reasoning” in models like OpenAI’s o-series and DeepSeek-R1 — and the training of agentic systems that use tools and act over many steps — is reinforcement learning. If you want to understand where frontier AI is going, you need the RL toolbox.
Three learning paradigms#
| Paradigm | Signal | Example |
|---|---|---|
| Supervised | Labels: “the right answer is y” | ImageNet classification |
| Self-supervised | Structure: “predict the missing part” | Next-token prediction |
| Reinforcement | Evaluation: “that was worth +3” | Game playing, control, reasoning |
The key distinction: in RL nobody tells you what to do, only how well it went — often with delay. Learning from evaluation rather than instruction is what makes RL both powerful (no labels needed, can exceed the teacher) and hard (credit assignment, exploration).
The RL loop#
An agent takes actions in an environment, observes new states, and receives rewards. The goal: a policy (state → action mapping) that maximizes cumulative reward.
Why scientists should care#
- Control: DeepMind trained an RL controller for the magnetic coils of the TCV tokamak, shaping plasma configurations directly from sensor data.
- Discovery as search: AlphaTensor found faster matrix-multiplication algorithms; AlphaDev found faster sorting routines — RL over discrete design spaces.
- Adaptive experiments: steering a beamline, telescope, or sequencing run in response to incoming data is a sequential decision problem — exactly RL’s territory.
- Foundation models: RLHF and RL with verifiable rewards are how base LLMs become useful assistants and reasoning engines.
Honest caveat, up front: RL is sample-hungry, sensitive to reward design, and often the wrong tool when you do have labels. We return to “when not to use RL” in Section 5.