Week 13: Direct Preference Optimization and GRPO

Purpose of this lecture#

The RLHFReinforcement Learning from Human Feedback pipeline from Week 12 works: it produces well-aligned language models that score highly on human preference evaluations. It also carries a significant overhead. PPOProximal Policy Optimisation-based RLHFReinforcement Learning from Human Feedback requires four large neural networks loaded simultaneously — the active policy, a frozen reference model, a reward model, and a value critic. For a 70B parameter model in BF16, this represents roughly 560 GB of GPU memory before accounting for optimizer state and activations. The result is that full-scale RLHFReinforcement Learning from Human Feedback is accessible to only a small number of organizations, and iteration cycles are slow even for those who can afford it.

This lecture develops two modern approaches that reduce this overhead without abandoning the theoretical grounding of the RLHFReinforcement Learning from Human Feedback framework. Direct Preference Optimization (DPODirect Preference Optimization) rederives the RLHFReinforcement Learning from Human Feedback objective to show that the reward model is implicit in the policy ratio and can be eliminated entirely: the alignment problem reduces to binary classification on preference pairs. Group Relative Policy Optimization (GRPOGroup Relative Policy Optimisation) takes the opposite simplification: keep the RLReinforcement Learning loop but replace the learned value critic with an empirical group baseline, eliminating the critic network while preserving token-level credit assignment.

Both methods trace directly to the closed-form KL-regularized optimal policy derived in Week 12. Understanding the derivations — not just the final loss functions — is what separates principled use of these methods from treating them as black-box recipes.

DPODirect Preference Optimization: the derivation and landmark result#

Direct Preference Optimization (Rafailov et al., 2023) demonstrated a surprising insight: the RLHF alignment objective has a closed-form optimal policy, and substituting that policy into the Bradley-Terry preference model yields a loss function that depends only on policy ratios, never on an explicit reward model. The paper's central contribution was empirical: DPO matched or beat PPO-based RLHF on summarization and single-turn dialogue while being substantially simpler to train — no reward model fitting, no sampling from the LM during fine-tuning, and no significant hyperparameter tuning. The community's initial reception was enthusiastic: DPO's simplicity, stability, and efficiency made RLHF-scale alignment accessible to academic labs. Subsequent work exposed limitations: Dubois et al. (2024) showed DPO underperforms PPO on mathematical and code reasoning tasks where step-level verification is important. Studies of DPO's implicit reward also revealed overoptimization pathologies — the policy can diverge sharply from the reference model on out-of-distribution prompts, and the implicit reward is often uninterpretable. The field has since answered with a wave of DPO variants — SimPO, IPO, ORPO, CPO — each patching a specific failure mode, taken up in the next section.

Starting point#

Recall from Week 12 that the KL-regularized RLHFReinforcement Learning from Human Feedback objective:

has the closed-form optimal policy:

where $Z(x) = \sum_y \pi_{\text{ref}}(y|x)\exp(r(x,y)/\beta)$ is the intractable partition function.

Inverting for the implicit reward#

DPODirect Preference Optimization's key move is to rearrange this expression to express $r(x,y)$ in terms of the policy ratio, rather than expressing $\pi^*$ in terms of $r$:

The term $\beta \log Z(x)$ depends only on $x$, not on $y$. Since the Bradley-Terry preference model from Week 12 involves the difference of rewards:

the $\beta \log Z(x)$ terms cancel exactly when the reward difference is taken:

The DPODirect Preference Optimization loss#

Substituting into the Bradley-Terry log-likelihood, and replacing the optimal policy $\pi^*$ with the parameterized policy $\pi_\theta$ we are training:

This is a binary cross-entropy loss whose "logit" is the difference in log-probability ratios (policy log-probability minus reference log-probability) between the preferred and dispreferred responses, scaled by $\beta$. No reward model appears — the algebra has eliminated it, leaving it implicit in the ratio $\pi_\theta / \pi_{\text{ref}}$.

What DPODirect Preference Optimization is actually optimizing#

Taking the gradient of $\mathcal{L}_{\text{DPO}}$ with respect to $\theta$ and examining the update direction reveals the mechanism:

where $\Delta = \beta\big[\log\frac{\pi_\theta(y^w \mid x)}{\pi_{\text{ref}}(y^w \mid x)} - \log\frac{\pi_\theta(y^l \mid x)}{\pi_{\text{ref}}(y^l \mid x)}\big]$ is the logit from the loss, and $\hat{\sigma} = 1 - \sigma(\Delta)$ is large when the model currently gets the pair wrong ($\Delta$ near zero or negative) and near zero when it already predicts the preference correctly. DPODirect Preference Optimization therefore concentrates updates on the pairs where the model is currently confused — the analog of hard-example mining in metric learning.

$\beta$ is the KL penalty strength: large $\beta$ keeps $\pi_\theta$ close to $\pi_{\text{ref}}$ — conservative, with low risk of collapse; small $\beta$ permits large policy shifts, risking the loss of the reference model's desirable behaviors.

Practical implications#

DPODirect Preference Optimization reduces the RLHFReinforcement Learning from Human Feedback training setup from four networks to two: the trainable policy $\pi_\theta$ and the frozen reference $\pi_{\text{ref}}$. Training is stable supervised learning — no reward signal variance, no PPOProximal Policy Optimisation clipping, no advantage estimation. The preference dataset $\mathcal{D}$ can be collected once offline and reused, making DPODirect Preference Optimization a fully offline alignment method in the sense of Week 11.

DPODirect Preference Optimization variants and limitations#

Sequence-level credit assignment#

DPODirect Preference Optimization evaluates the log-probability of the entire sequence $y$:

This means a response that contains 499 excellent tokens followed by one harmful token and one that is consistently poor receive the same treatment: the sequence is labeled as preferred or dispreferred as a whole. There is no mechanism to attribute the preference to specific tokens within the sequence. This sparse credit assignment is DPODirect Preference Optimization's principal limitation relative to PPOProximal Policy Optimisation: it cannot identify and correct specific failure modes within a response.

SimPO#

Simple Preference Optimization (SimPO, Meng et al., 2024) modifies DPODirect Preference Optimization by removing the reference model and normalizing log-probabilities by sequence length. The derivation starts directly from the DPODirect Preference Optimization implicit reward recovered earlier, $r(x,y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$, and makes two substitutions:

(a) Drop the reference-model ratio and use the length-normalized log-probability directly as the implicit reward:

Because this reward no longer references $\pi_{\text{ref}}$ at all, there is nothing left to keep around — scoring a completion needs only a forward pass through $\pi_\theta$.

(b) Substitute $r_{\text{SimPO}}$ into the same Bradley-Terry pairwise loss form used to derive $\mathcal{L}_{\text{DPO}}$, $-\log \sigma\!\left(r_{\text{SimPO}}(x,y^w) - r_{\text{SimPO}}(x,y^l)\right)$, and add a target margin $\gamma$ that the reward gap must clear before the loss stops penalizing the pair. This gives exactly the boxed SimPO objective below.

where $\gamma > 0$ is a margin. SimPO targets two empirical failings of DPODirect Preference Optimization. The first is reference-model dependence: DPO's implicit reward is the ratio $\log \pi_\theta - \log \pi_{\text{ref}}$, and at inference no reference model is involved — so the ranking DPO optimizes can disagree with the generation likelihood the model actually uses. The second is length bias: DPO sums per-token log-probabilities, so longer responses accumulate higher scores regardless of quality. SimPO replaces the summed log-probability with the length-normalized average — exactly the metric used to rank candidates at generation time — and subtracts the margin $\gamma$, so a pair stops contributing loss only once the preferred response's average log-probability beats the dispreferred one by at least $\gamma$. Removing $\pi_{\text{ref}}$ is also the memory win: compared to a vanilla DPODirect Preference Optimization implementation, SimPO cuts run time by roughly 20% and GPU memory by about 10% (Meng et al., 2024). The cost is the theoretical anchor — without a reference model the loss no longer descends from the KL-regularized RLHF objective, so the distance the policy has traveled from its initialization is no longer bounded by a measurable KL term.

GRPOGroup Relative Policy Optimisation: eliminating the critic#

Group Relative Policy Optimization (GRPOGroup Relative Policy Optimisation; Shao et al., 2024, introduced in DeepSeekMath) takes the opposite simplification from DPO: retain the RLReinforcement Learning loop and an explicit reward signal, but eliminate the learned value critic by replacing it with an empirical group baseline computed from multiple sampled completions. Shao et al. demonstrated that GRPO matches PPO performance on mathematical reasoning with significantly lower memory overhead (one critic network eliminated), enabling training of large reasoning models. The landmark result came with DeepSeek-R1 (2025): applied at scale with a rule-based verifier (correctness checking), GRPO elicited emergent chain-of-thought reasoning — models learned to generate extensive self-correcting solution traces without explicit supervision of the reasoning process itself, only outcome-level verification. This finding reignited the debate about process versus outcome reward models: if outcome-only rewards can implicitly elicit step-level reasoning behaviors, do process reward models (Lightman et al., 2023) provide additional value, or are they an unnecessary intermediate representation?

The critic's role and cost#

In PPOProximal Policy Optimisation (Week 7–8), the advantage estimate $A_t = Q(s_t, a_t) - V(s_t)$ requires a value function $V_\phi$ that predicts the expected return from each state. For a language model, $V_\phi$ is typically another copy of the language model with an added scalar head. This doubles the number of active parameters during training and requires a separate optimization loop to keep $V_\phi$ accurate.

The group baseline#

GRPOGroup Relative Policy Optimisation replaces the learned critic with an empirical baseline computed from a group of sampled completions. For each prompt $x$ in the training batch:

1. Sample $G$ completions $\{y_1, \ldots, y_G\}$ from the current policy $\pi_\theta$.
2. Score each with the reward model (or a verifier): $\{r_1, \ldots, r_G\}$.
3. Compute the group-normalized advantage for each completion:

The standardized score $A_i$ is positive for completions that outperform the group average and negative for those that underperform. It is a relative, not absolute measure of quality: a reward of 0.7 is a strong positive advantage if all other completions scored 0.3, but a negative advantage if others scored 0.9.

The GRPOGroup Relative Policy Optimisation objective#

The policy update applies the PPOProximal Policy Optimisation clipped surrogate objective using $A_i$ in place of the GAE advantage, adding a per-token KL penalty against the reference model:

where $\rho_{i,t} = \pi_\theta(y_{i,t} \mid x, y_{i,<t}) / \pi_{\text{old}}(y_{i,t} \mid x, y_{i,<t})$ is the per-token importance ratio. The KL penalty is computed token-by-token rather than at the sequence level, providing denser regularization.

Interactive: understanding GRPO advantage#

The core of GRPO is the relative scoring within a group. Consider a batch with $G = 4$ completions for the same prompt:

$\mu = 0.55$, $\sigma \approx 0.27$. Completions A and B are reinforced; C and D are suppressed. Now suppose we change only completion D's reward from 0.2 to 0.8 while keeping A's reward at 0.9:

Key insight: Completion A's reward stayed at 0.9, but its advantage fell from +1.30 to +1.05: raising D from 0.2 to 0.8 lifted the group mean from 0.55 to 0.70, so A's lead over the group narrowed even though A itself did not change. D flips from the worst completion (−1.30) to second-best (+0.53). The advantage is a relative, not absolute, measure — it depends on all completions in the group, not just the individual reward. This relative nature is why GRPO creates strong contrastive gradients for reasoning tasks: correct solutions look better when directly compared with incorrect solutions from the same prompt.

Why GRPOGroup Relative Policy Optimisation works for reasoning tasks#

The group sampling structure makes GRPOGroup Relative Policy Optimisation particularly effective for tasks with verifiable correct answers: mathematics, formal proofs, code execution, and structured reasoning. In these settings:

• The reward signal is binary or near-binary (correct / incorrect), making reward models unnecessary — a rule-based verifier suffices.
• Multiple completions per prompt are natural: the model generates several candidate solutions, and the verifier checks each. Correct solutions have $A_i > 0$; incorrect ones have $A_i < 0$.
• The contrastive signal between correct and incorrect solutions in the same group is a strong learning signal: the model simultaneously reinforces correct reasoning paths and suppresses incorrect ones from the same prompt.

This is the structure that enabled DeepSeek-R1's reasoning capabilities: by training on mathematical problems with verifiable answers using GRPOGroup Relative Policy Optimisation, the model learned to generate extended, self-correcting chain-of-thought reasoning — a behavior that emerges from the reinforcement signal rather than being explicitly supervised.

Credit assignment: sequence-level vs token-level#

The sequence-level vs token-level distinction is a fundamental axis along which alignment methods differ, and it has direct implications for what behaviors the model can and cannot learn.

Sequence-level (DPODirect Preference Optimization)#

The preference label is applied to the entire response. The gradient signal is a uniform rescaling of token log-probabilities throughout the response by the same factor. Every token in a preferred response is equally reinforced; every token in a dispreferred response is equally suppressed. This is appropriate when the preference is a global property of the response (style, tone, factual accuracy uniformly distributed) but breaks down when the preference is driven by a specific failure — a hallucinated fact in token 50, an unsafe statement in token 120 — embedded in an otherwise acceptable response.

Token-level (PPOProximal Policy Optimisation, GRPOGroup Relative Policy Optimisation)#

The advantage can vary across tokens within a single response. A critic-based advantage (PPOProximal Policy Optimisation) lets different positions receive different signals, since the GAE advantage already varies over time. GRPOGroup Relative Policy Optimisation's advantage $A_i$ is shared across the tokens of completion $i$ — what varies per position is the importance ratio $\rho_{i,t}$ and the per-token KL penalty, so different tokens still contribute differently to the update. In practice, PPOProximal Policy Optimisation's token-level credit assignment enables it to learn to avoid specific failure modes within responses in ways that DPODirect Preference Optimization cannot, at the cost of requiring either a learned critic (PPOProximal Policy Optimisation) or a group sampling overhead (GRPOGroup Relative Policy Optimisation).

Synthesis: choosing an alignment method#

The practical choice among these methods depends primarily on two factors: compute budget and task structure. DPODirect Preference Optimization is the default choice when the preference dataset is high-quality and the alignment target is a global property of responses (helpfulness, harmlessness). GRPOGroup Relative Policy Optimisation is preferred when the task has verifiable rewards and the goal is to train reasoning capabilities that require credit assignment at the reasoning-step level. PPOProximal Policy Optimisation remains the most principled option when the reward model is reliable, the task requires fine-grained credit assignment, and compute is available.

Exercise: comparing DPO and GRPO#

1. Analytical: Derive the SimPO loss from the DPO loss by applying sequence-length normalization.
2. Implementation: Given the GRPO advantage formula $A_i = (r_i - \mu)/\sigma$ for a group of $G$ completions, compute the advantages for the following group: $\{r\} = [0.95, 0.88, 0.45, 0.42, 0.15]$. Now add a KL penalty term: modify the advantage to $A_i^{\text{KL}} = (r_i - \mu)/\sigma - \beta \cdot \text{KL}_i$ where $\text{KL}_i$ is the per-completion KL divergence from the reference model. If $\beta = 0.1$ and the KL values are $[0.05, 0.12, 0.03, 0.02, 0.08]$, recompute the advantages. Which completions change sign, and why?
3. Reasoning: Explain why GRPO is inherently more robust to reward model hallucination than PPO when using a rule-based verifier.

Browser lab: DPO margin vs GRPO group baseline#

1. DPO: push $\log\pi(y_w)-\log\pi(y_l)$ above the reference margin (sigmoid loss).
2. GRPO-style: within a group of samples, advantage = reward − group mean (no critic).

import numpy as np

rng = np.random.default_rng(0)

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -40, 40)))

# --- DPO on one preference pair (scalar log-probs) ---
beta = 0.5
log_ref_w, log_ref_l = -2.0, -2.2
log_pi_w, log_pi_l = -2.0, -2.2
lr = 0.4
print("DPO steps (raise y_w over y_l):")
print(f"{'step':>4}  {'loss':>8}  {'Δ_θ - Δ_ref':>12}  {'σ':>6}")
for step in range(10):
    delta = beta * ((log_pi_w - log_ref_w) - (log_pi_l - log_ref_l))
    loss = -np.log(sigmoid(delta) + 1e-12)
    s = sigmoid(delta)
    g = beta * (1 - s)
    log_pi_w += lr * g
    log_pi_l -= lr * g
    if step % 2 == 0:
        print(f"{step:4d}  {loss:8.4f}  {delta/beta:12.3f}  {s:6.3f}")

# --- GRPO-style group relative advantages ---
# Group of G samples with proxy rewards; advantage A_i = r_i - mean(r)
G = 8
r = rng.normal(0.5, 1.0, size=G)
r[0] += 2.0  # one clearly better completion
A = r - r.mean()
# token-free: update log π_i ∝ A_i (group-normalized)
log_pi = np.zeros(G)
log_pi += 0.3 * A
z = log_pi - log_pi.max()
pi = np.exp(z); pi /= pi.sum()
print("\nGRPO-style group (no value network):")
print(f"  rewards: {np.round(r, 2)}")
print(f"  advantages: {np.round(A, 2)}")
print(f"  π after 1 step: {np.round(pi, 3)}  (mass on best={pi[0]:.3f})")
print("Notice: DPO needs paired prefs; GRPO needs only group rewards — critic-free relative signal.")

What to try: set beta=0.1 (weaker DPO) or G=4 with more reward noise.

Key takeaways#

DPODirect Preference Optimization rederives the RLHFReinforcement Learning from Human Feedback alignment problem by inverting the closed-form KL-regularized optimal policy to express the reward as a ratio of policy to reference log-probabilities. Substituting this into the Bradley-Terry preference model yields a binary cross-entropy loss on preference pairs that requires no reward model, no RLReinforcement Learning loop, and only two networks. The $\beta$ parameter plays the role of the KL penalty in RLHFReinforcement Learning from Human Feedback. DPODirect Preference Optimization's limitation is sequence-level credit assignment: it cannot attribute preference outcomes to specific tokens within a response. GRPOGroup Relative Policy Optimisation retains the RLReinforcement Learning loop and explicit reward signal but replaces the learned value critic with a group-normalized empirical baseline, eliminating the critic network. The contrastive structure of group sampling makes GRPOGroup Relative Policy Optimisation particularly effective for reasoning tasks with verifiable answers. SimPO extends DPODirect Preference Optimization by removing the reference model and adding length normalization. The sequence-level vs token-level credit assignment axis is the principal theoretical distinction among these methods, with practical implications for which failure modes each can address.

Conceptual questions#

1. Derive the DPODirect Preference Optimization loss function from scratch: start from the KL-regularized RLHFReinforcement Learning from Human Feedback objective, write down its closed-form optimal policy, invert to express $r(x,y)$ in terms of the policy ratio, and substitute into the Bradley-Terry preference model. At what step do the intractable partition functions $Z(x)$ cancel, and why is this cancellation exact? What property of the Bradley-Terry model enables it?

2. The DPODirect Preference Optimization gradient scales each preference pair update by a factor $\hat{\sigma} = 1 - \sigma(\Delta)$ where $\Delta$ is the log-ratio difference. For a pair where the model already correctly predicts the preference with high confidence ($\Delta \gg 0$), $\hat{\sigma} \approx 0$. Explain why this weighting is beneficial (analogous to hard example mining) and identify a failure mode it could cause if the training dataset contains mislabeled preference pairs.

3. GRPOGroup Relative Policy Optimisation samples $G$ completions per prompt and standardizes their rewards to compute advantages. If $G = 1$, what does the advantage equal, and what does the GRPOGroup Relative Policy Optimisation gradient become? If all $G$ completions receive identical rewards (e.g., all correct or all incorrect), what happens to the gradient? What does this imply about the minimum dataset and sampling conditions required for GRPOGroup Relative Policy Optimisation to produce a useful signal?

4. A model trained with DPODirect Preference Optimization on a preference dataset where the preferred response is always longer than the dispreferred response will develop a length bias. Explain mechanistically why this happens using the sequence-level log-probability sum, and show how SimPO's length normalization resolves it. Does PPOProximal Policy Optimisation-RLHFReinforcement Learning from Human Feedback suffer from the same bias? Why or why not?

5. DeepSeek-R1 uses GRPOGroup Relative Policy Optimisation with a rule-based verifier (checking mathematical correctness) rather than a learned reward model. Compare this to the standard RLHFReinforcement Learning from Human Feedback reward model approach from Week 12. Under what conditions does a rule-based verifier provide a more reliable training signal than a learned RM, and what category of tasks is structurally excluded from rule-based verification? For a task that cannot use rule-based verification, describe how you would design the reward signal and why GRPOGroup Relative Policy Optimisation may still be preferable to PPOProximal Policy Optimisation.

Knowledge Check#

Check DPO and GRPO design ideas.

Looking ahead#

Weeks 12–13's aligned models (RLHF/DPO reward models as task specs), Week 10's planning algorithms (MCTS, value-guided search over discrete tool calls), and the foundations (Week 1's MDP, Week 9's belief states over context windows, Week 4's TD credit assignment over long traces) stop being separate chapters in Week 14: the reward model is the agent's objective, the planner is its policy improvement operator, and the belief state is its context window. The capstone's job is to show the three collapsing into one deployed loop — and to name what still breaks at the joints (credit over 50-step traces, exploration with irreversible side effects).

Week 14: Agentic Systems. We examine how tool-using LLMs instantiate the MDPMarkov Decision Process formalism from Week 1, how hierarchical and compositional task structures map onto RLReinforcement Learning sub-problems developed throughout the course, and what the remaining open problems in agentic AI are from a reinforcement learning perspective.

Further reading#

• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS. (DPODirect Preference Optimization).
• Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv. (Introduced GRPOGroup Relative Policy Optimisation).
• Meng, Y., et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv. (SimPO).