Wrap-Up & Resources#

(Lecture time: 1:20–1:30, incl. Q&A)

The whole lecture on one slide#

  • RL = learning from evaluation, not instruction.
  • The Bellman equation is the recursive backbone; everything else approximates it.
  • Deep RL = replace tables with networks; DQN stabilizes value learning, PPO stabilizes policy learning.
  • LLMs are policies. RLHF aligned them; RLVR (verifiable rewards + GRPO-style training) produced reasoning models; agents are the open frontier.
  • In science, RL shines for control, design search, and adaptive experimentation — and is the wrong tool when you have labels, no simulator, or no clean reward.

The whole lecture on one map

Where to go next#

Books & courses

  • Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.) — free online, still the canonical text.
  • OpenAI Spinning Up in Deep RL — the best bridge from theory to code.
  • Hugging Face Deep RL Course — hands-on, free, modern tooling.
  • David Silver’s UCL lectures — classic video series for the foundations.

Code

  • CleanRL — single-file, readable implementations of the classic algorithms.
  • Gymnasium — standard environments (CartPole, LunarLander, Atari, MuJoCo).
  • TRL (Hugging Face) and verl — RLHF/GRPO training for LLMs.
  • Stable-Baselines3 — batteries-included classic deep RL.

Papers to start with (one per lecture section)

  1. Mnih et al., Human-level control through deep RL (DQN, Nature 2015)
  2. Schulman et al., Proximal Policy Optimization (2017)
  3. Ouyang et al., Training LMs to follow instructions with human feedback (InstructGPT, 2022)
  4. DeepSeek-AI, DeepSeek-R1: Incentivizing reasoning capability in LLMs via RL (2025)
  5. Degrave et al., Magnetic control of tokamak plasmas through deep RL (Nature 2022)

Contact#

Elynn Chen · elynn.chen@stern.nyu.edu · elynncc.github.io.

Slides and the demo notebook are linked from the landing page.

Backward 5. RL in Science