The Policy-Gradient Zoo, Organized#

Companion to Policy-Based Deep RL and the mathematics page. The zoo of named algorithms compresses to one skeleton, five design knobs, and six families — and each family leans on its own small kit of mathematical tools. Numbered results (Theorem 1, Lemma 3, …) refer to the mathematics page.

One skeleton, five knobs#

Every method below is a variant of the same update,

specialized by five nearly-orthogonal choices:

1. Weight — what multiplies the score: $G(\tau)$, $G_t$, $G_t - b(s_t)$, critic advantage, GAE($\lambda$), group-standardized reward, reward-model score.
2. Gradient route — through the density (score-function) or through a value model (pathwise).
3. Restraint — none, natural-gradient preconditioning, KL constraint, clip, KL-to-reference penalty.
4. Data regime — strictly on-policy, stale-with-ratios, off-policy replay, offline.
5. Critic — none, $V_\phi$, $Q_\phi$, twin $Q$, soft $Q$, group mean, reward model.

The families below are the clusters that appear when these knobs are set coherently.

Family 1 — Monte-Carlo score-function methods#

Members. REINFORCE; REINFORCE with baseline; leave-one-out / RLOO-style multi-sample variants.

Design. Weight = sampled return (possibly centered); no critic; on-policy; no restraint.

Tool kit.

• Score-function identity $\nabla p_\theta = p_\theta \nabla \log p_\theta$ and differentiation under the integral (dominated convergence; support independent of $\theta$) — Theorem 1.
• Conditional-mean-zero scores, $\mathbb{E}[\nabla \log \pi_\theta(a|s) \mid s] = 0$ — the one fact powering both causality (Corollary 2) and baseline unbiasedness (Lemma 3).
• Control variates from classical Monte Carlo: baselines are exactly this, and the variance-optimal baseline is a score-weighted value.
• Robbins–Monro stochastic approximation — why following unbiased noisy gradients converges to stationary points at all.

Signature failure justified against: variance. Everything here is unbiased; nothing is fast.

Family 2 — Actor–critic methods#

Members. A2C / A3C; GAE-based actor–critic; natural actor–critic.

Design. Weight = bootstrapped advantage from a learned $V_\phi$; on-policy; usually no explicit restraint.

Tool kit.

• Bellman equations as consistency conditions: the TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ has conditional mean $A^\pi(s_t,a_t)$ at the true $V^\pi$ — the license to use $\delta_t$ as an advantage estimate (and, more deeply, a martingale characterization of $V^\pi$).
• Geometric-series resummation: GAE is the $\lambda$-resolvent $\sum_j (\gamma\lambda)^j \delta_{t+j}$; the same $(I - \gamma P)^{-1}$ algebra that underlies all discounted RL.
• Compatible function approximation (Sutton et al., 2000): for critics of a matched linear form, the policy gradient computed with the approximate critic is exact — the classical answer to “doesn’t a wrong critic bias everything?”
• Two-timescale stochastic approximation (Borkar): convergence arguments run the critic on a fast clock and the actor on a slow one, so the actor sees a quasi-stationary critic.

Signature failure justified against: variance again — now paying with bias, priced by $\lambda$.

Family 3 — Trust-region and proximal methods#

Members. Natural policy gradient; TRPO; PPO (clip and KL-penalty variants).

Design. Weight = GAE; restraint is the defining feature; data may be reused for a few epochs via ratios.

Tool kit.

• Performance difference lemma (Lemma 4) — the exact identity that makes policy comparison possible from one policy’s data.
• Change of measure / importance sampling — the ratio $\pi'(a \mid s)/\pi(a \mid s)$ corrects actions; the impossibility of the state ratio $\rho^{\pi'}/\rho^{\pi}$ is why the surrogate exists.
• First-order agreement of the surrogate — same value, same gradient at $\pi' = \pi$, with second-order error, both consequences of zero-mean advantage (see the remark on the mathematics page).
• Monotonic improvement bound (Theorem 5) with Pinsker’s inequality converting total variation to KL.
• Information geometry: natural gradient preconditions by the inverse Fisher matrix — steepest ascent measured in KL, not in parameter space; the modern optimization reading is mirror descent / proximal point.
• For PPO specifically: no improvement theorem; three provable local properties of the clipped objective substitute (mathematics page, §5), and the guarantee-to-practice gap is documented (Engstrom et al., 2020).

Signature failure justified against: destructive updates — a bad step corrupts the next batch of data.

Family 4 — Deterministic / pathwise methods#

Members. DPG; DDPG; TD3.

Design. No score, no density: gradient flows through the critic. Off-policy replay is native. Continuous actions only.

Tool kit.

• Deterministic policy gradient theorem (Silver et al., 2014) — the chain rule through $Q$: $\nabla_\theta J = \mathbb{E}[\nabla_\theta \mu_\theta(s)\, \nabla_a Q(s,a)\vert_{a=\mu_\theta(s)}]$; recoverable as the $\sigma \to 0$ limit of the stochastic theorem.
• Pathwise derivatives — the same estimator family as infinitesimal perturbation analysis in simulation optimization; contrasts with the measure-change family above.
• Off-policy legitimacy without ratios: no action distribution, hence nothing to importance-correct — the formal reason replay is free here.
• Optimizer’s-curse corrections: the actor climbs an estimated critic, so value noise is selected upward; TD3’s twin-$Q$ minimum is a sample-splitting-style fix (the same statistics as Double DQN).
• Caveat inherited from value learning: bootstrapped critics under function approximation live inside the deadly triad; target networks are the standard damping.

Signature failure justified against: sample cost (replay) — paying with critic exploitation risk.

Family 5 — Maximum-entropy / inference-flavored methods#

Members. SAC; soft Q-learning; the “RL as probabilistic inference” line.

Design. Stochastic policy, trained pathwise via reparameterization; off-policy replay; entropy bonus as the restraint-like ingredient.

Tool kit.

• Gibbs variational principle: $\max_\pi \mathbb{E}_\pi[r] + \beta\,\mathcal{H}(\pi)$ has Boltzmann solution $\pi^* \propto e^{Q/\beta}$; log-sum-exp is the soft max — free-energy algebra from statistical mechanics.
• Reparameterization trick ($a = \mu_\theta(s) + \sigma_\theta(s)\,\epsilon$): a pathwise gradient through a stochastic policy — the bridge between Families 1 and 4.
• Soft Bellman operators: contraction arguments survive the entropy augmentation, so soft policy iteration has clean fixed-point theory.
• Duality: entropy-regularized control is Legendre-dual to log-partition computation; this is the same identity DPO exploits one family below.

Signature failure justified against: brittle exploration and hyperparameter sensitivity — entropy keeps the policy from collapsing early.

Family 6 — LLM post-training methods#

Members. RLHF with PPO; DPO; RLVR with GRPO (and kin: RLOO, REINFORCE-style LLM trainers).

Design. Policy = language model (softmax over tokens, autoregressive); score-function route (discrete actions close the pathwise door); restraint = clip

• KL-to-reference; critic = reward model, or a group mean, or a verifier.

Tool kit.

• Bradley–Terry / paired-comparison models (psychometrics, 1952) — the reward-model likelihood.
• KL-regularized control in closed form: the Gibbs solution $\pi^*(y \mid x) \propto \pi_{\mathrm{ref}}(y \mid x)\, e^{r(x,y)/\beta}$ — read forward it justifies the RLHF objective; read backward (solve for $r$ in terms of $\pi^*$) it is DPO.
• Control variates again: GRPO’s group mean is Lemma 3 with a Monte-Carlo, per-prompt baseline — no critic network, $K$ samples instead.
• Weighted cross-entropy identity: the surrogate is SFT on self-generated tokens, weighted by score (derived on the main page) — the formal bridge between pre-training-style losses and RL.
• Goodhart / reward-hacking caveats: optimizing against a learned proxy is where the KL anchor earns its keep; with verifiable rewards (RLVR) the proxy risk drops and optimization can push harder.

Signature failure justified against: proxy exploitation and distribution drift from the pretrained reference.

The grid#

Read rows to reconstruct algorithms; read columns to see that most “new” methods change exactly one cell.

In what sense is any of this correct? Three layers, weakening as the algorithms get fancier: (i) for exact weights the update is the gradient — Theorem 1 plus the covariance reading ($\mathbb{E}[c \cdot \nabla\log\pi] = 0$, so only reward–action correlation pushes); (ii) unbiased noisy gradients suffice by Robbins–Monro — the same guarantee as all of SGD; (iii) biased variants (critics, GAE, clipping) replace exactness with controlled perturbation: compatible function approximation, first-order agreement, Theorem 5's bound. Nothing at any layer promises global optima, and with deep networks layer (iii) is a guide, not a theorem.

References beyond the other pages#

• Borkar (2008), Stochastic Approximation: A Dynamical Systems Viewpoint — two-timescale analysis.
• Haarnoja et al. (2017–18), soft Q-learning and SAC; Levine (2018), RL and control as probabilistic inference: tutorial and review.
• Rafailov et al. (2023), Direct preference optimization — the closed-form KL-regularized solution read backward.
• Ahmadian et al. (2024), Back to basics: revisiting REINFORCE-style optimization for LLMs — RLOO and the case for simple score-function baselines at scale.
• Everything else: see the reference lists on the main page and the mathematics page.