RL in Science#

(Lecture time: 1:10–1:20)

Three case studies spanning the main archetypes: continuous control, discrete design search, and adaptive experimentation.

Case 1 — Continuous control: tokamak plasma shaping#

DeepMind + EPFL (Nature, 2022) trained a deep RL controller for the 19 magnetic coils of the TCV tokamak. The controller maps magnetic sensor measurements directly to coil voltages at control frequency, and was trained entirely in a simulator before deployment on the real machine, holding and shaping plasma configurations — including ones that were laborious to achieve with classical controllers.

Archetype: high-dimensional continuous control, sim-to-real transfer, actor-critic training. Lesson: a good simulator is the real asset; RL converts simulator access into a controller.

Case 2 — Discrete design search: AlphaTensor & AlphaDev#

Frame “find a better algorithm” as a single-player game: states are partial constructions, actions extend them, reward is (negative) cost of the finished object.

• AlphaTensor (2022): found matrix-multiplication algorithms beating Strassen’s 49-multiplication record for 4×4 matrices in $\mathbb{F}_2$.
• AlphaDev (2023): found shorter assembly sorting routines, merged into LLVM’s libc++.

Archetype: MuZero-style search + self-play over combinatorial spaces. Lesson: if you can score a candidate design cheaply and exactly, RL-with-search can explore design spaces no human enumerates.

Case 3 — Adaptive experimentation: steering the instrument#

Very on-brand for a DOE lab: beamline alignment and tuning, adaptive scan strategies (spend dwell time where information is), telescope scheduling, self-driving chemistry labs. Each is a sequential decision problem where the “environment” is your instrument and the reward is information gained or samples-to-target.

Archetype: bandits / Bayesian optimization shading into full RL as the horizon grows. Lesson: you may already be doing RL manually; formalizing it lets you automate and improve it.

When not to use RL#

Be the person in your lab who asks these questions before the six-month RL project:

• Do you have labels? Then supervised learning is simpler, cheaper, and stronger.
• Is one-shot optimization enough? Bayesian optimization or evolutionary methods beat RL when there's no sequential structure.
• Can you afford the samples? Model-free RL routinely needs millions of interactions. No fast simulator ⇒ big problem.
• Can you specify the reward? Misspecified rewards get exactly what you asked for — reward hacking is Goodhart's law with a gradient.
• Sim-to-real: a policy is only as good as the simulator it was trained in.