Policy-Based Deep RL: REINFORCE to PPO#

Any differentiable distribution head bolted onto a network: the body maps $s$ to features, the head turns features into a distribution over actions. The policy-gradient machinery only ever asks for one thing — a tractable $\log \pi_\theta(a \mid s)$ and its gradient. In practice:

• Softmax over discrete actions — the network outputs logits $z_a$; then $\log \pi_\theta(a|s) = z_a - \log \sum_{a'} e^{z_{a'}}$, numerically stable and identical to a classification head. This is the case for games — and, crucially, for LLMs: each token is one action sampled from a softmax over the vocabulary, so a language model is already a policy in exactly this sense, and the REINFORCE surrogate becomes a return-weighted cross-entropy.
• Gaussian policies — the network outputs $\mu_\theta(s)$ and (log-)variance: $\pi_\theta(a|s) = \mathcal{N}\big(\mu_\theta(s), \Sigma_\theta(s)\big)$. The log-density is a quadratic in $a$, so the surrogate is a return-weighted squared error pulling $\mu_\theta(s)$ toward actions that worked. Standard for continuous control (torques, voltages); actions are squashed by $\tanh$ when bounded.
• Everything else with a differentiable log-density — Beta distributions for naturally bounded actions, mixtures for multimodal ones, autoregressive factorizations $\prod_i \pi_\theta(a_i \mid a_{<i}, s)$ for high-dimensional or sequential action spaces (an LLM response is exactly this), normalizing flows when extra flexibility is needed.

Two boundary notes. The math never requires softmax — the log-derivative identity below is just the chain rule and holds for any of these families. And the policy must be genuinely stochastic: a deterministic policy has no density to take the log of, which is why deterministic continuous control (DDPG and kin) rests on a different theorem that differentiates through $Q$ instead.

Why the score-function estimator dies at determinism. A deterministic policy $a = \mu_\theta(s)$ is a degenerate distribution — a point mass — with no density to take the log of, so $\nabla_\theta \log \pi_\theta$ does not exist. And the boundary is not merely formal: watch the Gaussian case approach it. Shrink $\sigma \to 0$ in the Gaussian surrogate and the gradient's magnitude scales like $1/\sigma^2$ while the exploration that feeds it vanishes — the estimator's variance blows up precisely as the policy becomes deterministic. REINFORCE needs randomness the way finite differences need a step size.

The other theorem. If actions are continuous and we have a critic $Q(s,a)$ that is differentiable in $a$, we can bypass densities entirely and push the gradient through the critic by the chain rule. The deterministic policy gradient (DPG, Silver et al., 2014):

$$ \nabla_\theta J = \mathbb{E}_{s \sim \rho^\mu} \Big[ \nabla_\theta\, \mu_\theta(s)\; \nabla_a Q^\mu(s, a)\big|_{a = \mu_\theta(s)} \Big]. $$

Read it as a two-stage chain rule: the critic's action-gradient $\nabla_a Q$ says "locally, which direction in action space increases value"; the actor Jacobian $\nabla_\theta \mu_\theta(s)$ translates "move the action that way" into "move the parameters this way." Note the reversal of demands relative to REINFORCE: the score-function estimator never differentiates the reward or the critic but needs a stochastic policy; DPG never needs a density but must differentiate a learned $Q$ — so it is exclusive to continuous actions, and its gradient is only as good as the critic it differentiates.

What determinism buys. No expectation over actions to estimate (one source of variance simply gone), and natural off-policy training: there is no action distribution to importance-correct, so learning from replayed data needs no ratios. This is why the DPG family, not REINFORCE, inherited the value-based page's toolkit.

DDPG (2016) is exactly that inheritance: DPG + replay buffer + target networks — "DQN for continuous actions," with the actor supplying the $\arg\max$ that continuous $Q$-learning could not afford. One new obligation: a deterministic policy explores nothing, so exploration must be bolted on externally (Gaussian or OU noise added to $\mu_\theta(s)$ during data collection).

TD3 (2018) names and fixes DDPG's characteristic failure. The actor is trained to climb $\nabla_a Q$, i.e. to seek out maxima of the estimated critic — and estimated maxima are optimistically wrong (the optimizer's curse, third appearance in these notes). DDPG therefore exploits its own critic's errors. TD3's three countermeasures: clipped double-Q (train two critics, use the minimum as the target), delayed policy updates (let the critic settle before the actor moves), and target-policy smoothing (add noise to target actions so the critic cannot be sharp at spurious peaks).

The bigger map: two gradient families. REINFORCE-style (likelihood-ratio / score-function) gradients differentiate $\log \pi_\theta$ and treat the reward as a black box; DPG-style (pathwise) gradients differentiate through a value model. SAC sits instructively in between: a stochastic policy trained pathwise, via the reparameterization trick ($a = \mu_\theta(s) + \sigma_\theta(s)\,\epsilon$ lets gradients flow through the sample). And the map explains the LLM chapter's choice for us: tokens are discrete, so there is no pathwise route through the sampling step — which is why RLHF and GRPO are built on the score-function family, the one that never needed to differentiate through anything.

The identity is one line: The chain rule applied to the logarithm: $$ \nabla_\theta \log p_\theta(x) = \frac{\nabla_\theta\, p_\theta(x)}{p_\theta(x)} \quad\Longleftrightarrow\quad \nabla_\theta\, p_\theta(x) = p_\theta(x)\, \nabla_\theta \log p_\theta(x), $$ is valid for any family of distributions, discrete or continuous, so long as $p_\theta(x) $ is differentiable in $\theta $ and strictly positive at the point where you evaluate it (you're dividing by it). Applying $\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta$,

$$ \nabla_\theta \mathbb{E}_{x \sim p_\theta}[f(x)] = \mathbb{E}_{x \sim p_\theta}\big[ f(x)\, \nabla_\theta \log p_\theta(x) \big]. $$

Applied to trajectories, $p_\theta(\tau) = p(s_0) \prod_t \pi_\theta(a_t|s_t) P(s_{t+1}|s_t,a_t)$, and the dynamics terms do not depend on $\theta$ (so they vanish from the gradient). That is the quiet miracle: the policy gradient requires no knowledge of, and no differentiation through, the environment. $f$ need not be differentiable either, i.e., returns can come from a simulator, a verifier, or a human rating. This is precisely why the same machinery will transfer to LLM training, where the "environment" is a reward model or a unit test.

The price: this is a score-function (likelihood-ratio) estimator, and its variance is notoriously large. The entire next act of this page is variance management. Two standard reductions come almost free: causality (action $a_t$ cannot influence rewards before $t$, so weight by the return-to-go $G_t$, not the total return) and baselines, next.

Detailed proofs in Corollary 2 on the mathematics page.

Take discrete actions with a softmax policy: the network outputs logits $z = f_\theta(s) \in \mathbb{R}^{|\mathcal{A}|}$ and

$$ \pi_\theta(a \mid s) = \frac{e^{z_a}}{\sum_{a'} e^{z_{a'}}}, \qquad \log \pi_\theta(a \mid s) = z_a - \log \sum_{a'} e^{z_{a'}}. $$

Step 1 — recall the classification loss. Cross-entropy for a one-hot label $y = a^\star$ on the same softmax is $\mathrm{CE}(z, a^\star) = -\log \pi_\theta(a^\star \mid s)$: "maximize the log-probability of the correct class."

Step 2 — write the REINFORCE surrogate. The sampled policy gradient says: change $\theta$ in the direction $\sum_t \hat{G}_t \, \nabla_\theta \log \pi_\theta(a_t \mid s_t)$. The loss whose negative gradient is exactly this direction is $$ \mathcal{L}(\theta) = - \sum_t \hat{G}_t \, \log \pi_\theta(a_t \mid s_t) = \sum_t \hat{G}_t \; \mathrm{CE}\big(z_t,\, a_t\big), $$ where the target treats each sampled action $a_t$ as if it were the correct label ($\hat G_t$ is treated as a constant — no gradient flows through it). That is the claim: cross-entropy toward your own samples, weighted by the return that followed. Nothing else in the training stack changes.

Step 3 — see what it does to the logits. Differentiate the cross-entropy $\mathrm{CE}(z, a_t) = -z_{a_t} + \log \sum_{a'} e^{z_{a'}}$ with respect to one logit $z_{a'}$. The first term contributes $-\mathbb{1}\{a' = a_t\}$; the log-sum-exp term contributes $e^{z_{a'}} / \sum_{a''} e^{z_{a''}} = \pi_\theta(a' \mid s_t)$ — the softmax reappears as its own gradient. Together, componentwise and in vector form:

$$ \frac{\partial\, \mathrm{CE}}{\partial z_{a'}} = \pi_\theta(a' \mid s_t) - \mathbb{1}\{a' = a_t\}, \qquad \nabla_z\, \mathrm{CE} = \pi_\theta(\cdot \mid s_t) - e_{a_t}, $$

the familiar "softmax minus one-hot" of classification. The REINFORCE surrogate just scales this by the return:

$$ \frac{\partial \mathcal{L}}{\partial z_{a'}} = \hat{G}_t \Big( \pi_\theta(a' \mid s_t) - \mathbb{1}\{a' = a_t\} \Big). $$

Reading it off, for $\hat G_t > 0$ (gradient descent on $\mathcal{L}$):

• The chosen action $a_t$: gradient $\hat{G}_t \big(\pi_\theta(a_t|s_t) - 1\big) \le 0$, so its logit moves up — by more when the action was unlikely ($1 - \pi_\theta(a_t|s_t)$ large). Surprising successes get the biggest updates.
• Every other action $a'$: gradient $\hat{G}_t\, \pi_\theta(a'|s_t) \ge 0$, so its logit moves down, in proportion to how much probability it currently holds.
• Conservation: the components sum to $\hat{G}_t (\sum_{a'} \pi_\theta(a'|s_t) - 1) = 0$ — the update only redistributes probability, consistent with softmax's invariance to shifting all logits by a constant.

With a baseline subtracted, a negative-advantage action flips every sign: its own logit is pushed down and its competitors up — "anti-classification." And the same $(\text{prob} - \text{one-hot})$ structure means one more thing for LLMs: per token, this gradient is identical in shape to the SFT gradient, differing only by the scalar weight — the code-level cash value of the SFT connection in the closing paragraph.

Why this framing earns its keep in Section 4: supervised fine-tuning of an LLM is this same loss with weight $1$ on demonstration tokens; REINFORCE-style RL is the same loss with weight $\hat G_t$ (or $\hat A_t$) on self-generated tokens. In one line: RL fine-tuning is SFT on your own samples, weighted by how well they scored. PPO and GRPO refine the weights and add the trust region — the loss's skeleton stays this one.

Softmax buys convenience, not validity: the policy-gradient theorem asks only for a differentiable $\log \pi_\theta(a \mid s)$, and the form of the surrogate loss is simply whatever that log-density looks like. Two worked cases:

Discrete / softmax. $\log \pi_\theta(a|s) = z_a - \log \sum_{a'} e^{z_{a'}}$ — logits minus log-sum-exp, numerically stable, and the surrogate becomes the return-weighted cross-entropy derived above.

Continuous / Gaussian. With $\pi_\theta(a|s) = \mathcal{N}\big(\mu_\theta(s), \sigma^2 I\big)$,

$$ -\log \pi_\theta(a \mid s) = \frac{\lVert a - \mu_\theta(s) \rVert^2}{2\sigma^2} + d \log \sigma + \text{const}, $$

so the REINFORCE surrogate $\mathcal{L} = -\sum_t \hat G_t \log \pi_\theta(a_t|s_t)$ is a return-weighted squared error: minimizing it pulls the mean $\mu_\theta(s_t)$ toward sampled actions that scored well, with strength $\hat G_t / \sigma^2$ — and, with a baseline, pushes it away from negative-advantage actions. Regression toward your own good samples, the exact continuous analogue of "cross-entropy toward your own good samples."

If $\sigma$ is learned, its gradient is interpretable too: minimizing $\hat G_t \,(-\log \pi_\theta)$ in $\sigma$ moves $\sigma$ toward $\lVert a_t - \mu_\theta(s_t) \rVert / \sqrt{d}$ for well-scoring actions — the policy widens when good outcomes keep arriving far from its current mean (there is more to explore out there) and narrows as they concentrate at the mean. Exploration adapts as a by-product of the same loss.

The general statement. Beta for bounded actions, mixtures for multimodal ones, autoregressive factorizations for sequences — plug in the log-density and the surrogate writes itself. What never changes: the estimator differentiates only $\log \pi_\theta$, never the reward and never the environment. Same theorem, different log-density.