Direct Preference Optimization#

The previous page ended with a bill — four transformer-scale networks — and a question: if we can write down the optimal policy, do we need the RL loop at all? DPO (Rafailov et al. 2023) answers no. In the deletion narrative, this page deletes the most: the reward model, the critic, and the entire sampling loop go at once, leaving a supervised-looking loss on a fixed dataset. The trick is a change of variables, and both facts it needs were planted on the RLHF page.

The two planted facts#

1. The Gibbs closed form. The KL-regularized objective is solved by $$ \pi^*(y \mid x) = \tfrac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\exp\!\big(\tfrac{1}{\beta} r(x, y)\big), $$ with an intractable per-prompt normalizer $Z(x)$.
2. Shift-invariance. The Bradley–Terry loss only ever sees reward differences on the same prompt, so any per-prompt constant added to $r$ is invisible.

The inversion#

Fact 1 maps rewards to policies. Read it backwards — take logs and solve for the reward:

Every reward function is expressible through its own optimal policy, up to a term that depends only on the prompt. Now substitute this into the Bradley–Terry model and watch fact 2 fire: the $\beta \log Z(x)$ term is a per-prompt constant, so it cancels in the difference,

The intractable object is gone, and the preference likelihood is written directly in terms of the policy. Doing maximum likelihood — the same estimation principle as the reward-model loss, on the same dataset — but over $\pi_\theta$ instead of $r_\phi$ gives the DPO loss:

The bracketed quantity $\hat{r}_\theta(x, y) = \beta \log \tfrac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$ is the implicit reward: DPO is exactly reward-model training, with the policy’s own log-ratio playing the role of the reward head. Hence the paper’s subtitle — your language model is secretly a reward model. Nothing about the objective changed; we changed which parameterization of it we do gradient descent in.

What the gradient does. Differentiating gives $$ \nabla_\theta \mathcal{L}_{\text{DPO}} = -\,\beta\, \mathbb{E}\Big[ \sigma\!\big(\hat{r}_\theta(x, y_l) - \hat{r}_\theta(x, y_w)\big) \big( \nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \big) \Big]: $$ push up the chosen response, push down the rejected one, with a weight that is large precisely when the implicit reward currently gets the pair wrong. A policy-gradient shape — score functions weighted by a reward-like scalar — with no sampling anywhere.

The code#

Given sequence log-probs from the Section 4.1 primitive (one masked sum per response, under the policy and under the frozen reference), the loss is a dozen lines:

import torch
import torch.nn.functional as F
torch.manual_seed(0)

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps,    ref_rejected_logps,    beta=0.1):
    """
    All inputs: (B,) sequence log-probs, log pi(y|x), from the Section 4.1
    primitive (masked sum of per-token log-probs).
    """
    chosen_rewards   = beta * (policy_chosen_logps   - ref_chosen_logps)    # implicit r-hat
    rejected_rewards = beta * (policy_rejected_logps - ref_rejected_logps)
    loss = -F.logsigmoid(chosen_rewards - rejected_rewards).mean()
    margin   = (chosen_rewards - rejected_rewards).detach()
    accuracy = (margin > 0).float().mean()
    return loss, chosen_rewards.detach(), rejected_rewards.detach(), accuracy

# Toy batch of 4 preference pairs: pretend sequence log-probs
policy_chosen   = torch.tensor([-14.2, -20.1,  -9.8, -31.0], requires_grad=True)
policy_rejected = torch.tensor([-15.0, -19.5, -12.4, -30.2], requires_grad=True)
ref_chosen      = torch.tensor([-14.6, -19.8, -10.9, -30.5])
ref_rejected    = torch.tensor([-14.9, -19.6, -11.8, -30.1])

loss, r_w, r_l, acc = dpo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected)
loss.backward()

print("loss:", round(loss.item(), 4))
print("implicit reward (chosen):  ", r_w.round(decimals=3))
print("implicit reward (rejected):", r_l.round(decimals=3))
print("reward accuracy:", acc.item())
print("grad wrt chosen log-probs: ", policy_chosen.grad.round(decimals=4))

loss: 0.6767
implicit reward (chosen):   tensor([ 0.0400, -0.0300,  0.1100, -0.0500])
implicit reward (rejected): tensor([-0.0100,  0.0100, -0.0600, -0.0100])
reward accuracy: 0.5
grad wrt chosen log-probs:  tensor([-0.0122, -0.0127, -0.0114, -0.0127])

Read the last line against the gradient formula: all entries are negative (chosen log-probs get pushed up), and the magnitudes are largest for pairs 2 and 4 — exactly the pairs whose implicit-reward margin is currently negative, i.e. the ones the model ranks wrongly. The theory’s weighting is visible in four numbers.

Two practical notes the code makes concrete: the batch’s reward accuracy — how often $\hat{r}_\theta$ ranks the pair correctly — is DPO’s standard training diagnostic; and the reference log-probs carry no gradient, so in real pipelines they are precomputed once over the dataset and the reference model never has to be in memory during training. The four-network bill of 4.2 drops to one.

What got deleted — and what it cost#

Deleted: the reward model (implicit now), the critic (no advantages to estimate), and generation itself — DPO never samples during training. What remains is minibatch gradient descent on a fixed dataset: RLHF at the cost of supervised fine-tuning.

The fine print: the derivation identifies the optimum of the KL-regularized objective, assuming the Bradley–Terry model is right and preferences are plentiful. It says nothing about the path gradient descent takes in the new parameterization, and the data is frozen — the method is offline in exactly the Section 2 sense, with everything that implies. The two deep dives below take these seriously.

Handoff#

DPO wins when the target is preference-shaped: subjective, pairwise, cheap to collect, expensive to verify. But 2024 turned attention to targets with the opposite shape — math answers, unit tests, formal proofs — where a program can grade any response exactly, and grading fresh on-policy samples costs nothing. For those, the field brought the RL loop back, minus the component that made it expensive: Section 4.4: verifiable rewards and GRPO.

References.

• Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model;
• Azar et al. (2023), A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO);
• Ethayarajh et al. (2024), KTO: Model Alignment as Prospect Theoretic Optimization;
• Meng et al. (2024), SimPO: Simple Preference Optimization with a Reference-Free Reward;
• Razin et al. (2024), Unintentional Unalignment: Likelihood Displacement in DPO;
• Pal et al. (2024), Smaug: Fixing Failure Modes of Preference Optimisation (DPO-Positive);
• Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment?;
• Tajwar et al. (2024), Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data.