Week 12: Reinforcement Learning from Human Feedback

Purpose of this lecture#

The entire theoretical apparatus developed in Weeks 1–11 — MDPs, value functions, policy gradients, actor-critics, PPOProximal Policy Optimisation, offline RLReinforcement Learning — was built around the assumption that the reward function is given. An agent playing Atari receives a score from the game engine; a robot receives a distance-to-goal signal from its simulator; a Q-learning agent on a gridworld receives $+1$ for reaching the goal.

For language models, this assumption fails. There is no function $r(x, y)$ that a programmer can write to evaluate whether a response $y$ to a prompt $x$ is helpful, honest, and harmless. The property being optimized is a human preference — a subjective, context-dependent judgment that cannot be reduced to a closed-form expression. Next-token prediction (cross-entropy on a text corpus) produces a fluent language model, but fluency and alignment are distinct: a model can be highly fluent while being sycophantic, misleading, harmful, or evasive.

Reinforcement Learning from Human Feedback (RLHFReinforcement Learning from Human Feedback) is the approach that bridges this gap: learn a reward model from human preference data, then optimize the language model against that reward model using RLReinforcement Learning. This lecture develops the full pipeline — SFT, reward modeling, KL-regularized PPOProximal Policy Optimisation — and connects each component to the RLReinforcement Learning theory developed throughout the course.

The three-stage RLHFReinforcement Learning from Human Feedback pipeline#

Stage 1: Supervised fine-tuning#

A base language model pretrained on web-scale text has learned to predict the statistical distribution of text on the internet — including low-quality, unhelpful, and unsafe text.

Supervised Fine-Tuning (SFT) trains the base model on a curated dataset of high-quality prompt–response pairs written or approved by human annotators:

SFT is standard next-token prediction on the curated dataset. Its purpose is not alignment but format shaping: after SFT, the model $\pi_{\text{SFT}}$ produces responses that look like helpful answers to prompts, rather than continuations of arbitrary web text. This provides a reasonable starting point for reward optimization and defines the reference distribution for the KL penalty in Stage 3.

The SFT model is also the foundation for the reward model: rather than training the RM from scratch, it is initialized from $\pi_{\text{SFT}}$.

Stage 2: Reward modeling#

The core challenge of RLHFReinforcement Learning from Human Feedback is learning $r_\phi(x, y)$: a function mapping a prompt–response pair to a scalar representing human preference. Two design choices — how preference data is collected and how it is modeled — make this tractable.

Pairwise comparisons over absolute scores

Human annotators are poor at assigning absolute quality scores (one annotator's 7 is another's 9), but are reliable at relative ranking: given two responses $y_1$ and $y_2$ to the same prompt $x$, annotators can consistently identify which is better. This shifts the learning problem from regression on absolute quality to classification on pairwise preferences.

This is the same move made by preference-based (dueling) bandits in Week 2: rather than observing an absolute reward for a single arm, the learner observes only which of two arms was preferred and must infer a scalar utility that explains the comparisons. Reward-model training in RLHFReinforcement Learning from Human Feedback is that same inference problem at language-model scale — a prompt–response pair plays the role of an arm, and the reward model $r_\phi$ is the latent utility function fit to pairwise judgments instead of absolute scores.

The preference dataset takes the form:

where $y^w_i \succ y^l_i$ indicates that the human preferred $y^w_i$ over $y^l_i$ for prompt $x_i$.

The Bradley-Terry preference model

The probability that a human prefers $y^w$ over $y^l$ is modeled as:

where $\sigma$ is the logistic sigmoid. This is the Bradley-Terry model (Bradley and Terry, 1952), originally developed for ranking sports teams. Its key property is that preferences are governed entirely by the difference in latent rewards — absolute reward scale is unidentified. This matches the human annotation setting: a human judges relative quality, and the model only needs to produce an ordering, not calibrated absolute scores.

The Bradley-Terry model's application to human preference learning assumes that human judgments form a total order (transitivity: if A > B and B > C, then A > C) that is consistent and independent of context. In practice, human preferences on complex reasoning tasks (math, code, scientific writing) are often intransitive, context-dependent, and irreproducible across annotators. The model's popularity in RLHF stems not from empirical validation of its assumptions but from mathematical convenience: it leads to a tractable maximum likelihood objective and has been validated post-hoc on simpler preference tasks: summarization quality (Stiennon et al., 2020) and harmlessness (Bai et al., 2022). For alignment tasks (honesty, helpfulness, safety), the gap between the model's assumptions and human judgment is substantial and remains underexplored.

The reward model is trained by maximum likelihood on the preference dataset:

This is binary cross-entropy: $y^w$ is the positive label and $y^l$ the negative, with logits given by the reward difference, so minimizing the loss pushes $r_\phi(x, y^w)$ above $r_\phi(x, y^l)$ for every pair in the dataset.

Reward model architecture

The RM is a copy of $\pi_{\text{SFT}}$ with the language model head replaced by a linear layer that outputs a scalar: it reads the full prompt–response pair as one sequence and returns $r_\phi \in \mathbb{R}$ at the final token position.

Stage 3: Policy optimization with PPOProximal Policy Optimisation#

With a reward model in hand, the LLMLarge Language Model optimization becomes a standard RLReinforcement Learning problem. The MDPMarkov Decision Process is defined as:

This is an episodic MDPMarkov Decision Process with a delayed terminal reward: the agent generates a full response $y$ token by token, and the reward model evaluates the complete response. The episode length is the response length $|y|$, which is variable.

The KL-regularized RLHFReinforcement Learning from Human Feedback objective

Optimizing $\pi_\theta$ directly against $r_\phi$ without constraint produces reward hacking: the policy finds responses that score highly under $r_\phi$ but are not actually preferred by humans. This is the extrapolation error from Week 11 — the RM was trained only on responses from the SFT model's distribution, so far from that distribution its predictions are unreliable and exploitable: the policy can discover out-of-distribution text patterns that game the RM's blind spots.

The fix is a KL divergence penalty anchoring the optimized policy to the SFT reference model:

where $\beta > 0$ controls the penalty strength. The KL term is computed token by token and summed over the response:

This is behavior regularization (Week 11) with $\pi_{\text{behavior}} = \pi_{\text{SFT}}$: RLHFReinforcement Learning from Human Feedback is offline RLReinforcement Learning on a preference dataset with $\pi_{\text{SFT}}$ as the behavior policy, and the KL term pins $\pi_\theta$ to it so the policy cannot stray into the OOD region where the reward model's signal is unreliable.

KL-regularized objective: the closed-form solution

The KL-regularized objective has an analytically tractable optimal policy. For any fixed reward function $r$, the solution to:

is the Boltzmann distribution:

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x,y)/\beta)$ is the partition function. This is the same Boltzmann optimal policy as in maximum entropy RLReinforcement Learning (Week 8, SACSoft Actor-Critic), with $\beta$ playing the role of temperature $\alpha$. The optimal policy concentrates on high-reward responses while maintaining support near the reference distribution. This closed-form solution is the starting point for deriving DPODirect Preference Optimization (Week 13), which bypasses the reward model entirely by solving for $r$ in terms of $\pi^*$ and $\pi_{\text{ref}}$.

PPOProximal Policy Optimisation in the RLHFReinforcement Learning from Human Feedback context

In practice, the KL-regularized objective is optimized with PPOProximal Policy Optimisation (Week 7–8). The token-level reward signal for PPOProximal Policy Optimisation is constructed as:

The terminal reward is the RM score minus the KL penalty at the final token; all intermediate steps receive only the per-token KL penalty. PPOProximal Policy Optimisation's clipped surrogate objective and GAE are then applied to this reward signal, with the language model serving as the actor and a separate value head (or a copy of the language model with an added scalar head) serving as the critic. This is the recipe Ouyang et al. (2022) used for InstructGPT: supervised fine-tuning, reward modeling, then PPOProximal Policy Optimisation against the RM lifted human preference ratings well above the SFT baseline, especially on instruction-following and harmful-output reduction. Their ablations did not isolate each stage's contribution, leaving open whether the reward-model stage is strictly necessary — a question Week 13's reward-model-free methods resolve by removing it.

The full RLHFReinforcement Learning from Human Feedback training loop therefore requires four networks in memory simultaneously: the policy $\pi_\theta$ (actor), the value function $V_\phi$ (critic), the SFT reference model $\pi_{\text{SFT}}$ (for KL computation), and the reward model $r_\phi$. The memory and compute demands of this four-network setup motivate the simpler preference optimization approaches in Week 13.

Limitations of vanilla RLHFReinforcement Learning from Human Feedback#

Overoptimization and Goodhart's Law#

The reward model $r_\phi$ is a proxy for true human preference, not the true preference itself. As PPOProximal Policy Optimisation pushes $\pi_\theta$ to maximize $r_\phi$, it eventually finds policies that score highly on the proxy while performing poorly on the true objective. This is Goodhart's Law: a measure used as a target ceases to be a good measure.

In practice, overoptimization produces characteristic failure modes: responses become verbosely padded to cover all angles (the RM rewards completeness), excessively hedged (the RM rewards safety language), or sycophantically agreeable (the RM rewards responses that match perceived user preferences). The gap between the proxy reward and true preference is often visualized by plotting RM score (increasing) against human evaluation score (increasing then decreasing) as a function of KL divergence from $\pi_{\text{SFT}}$ — the classic overoptimization curve (Gao et al., 2023). How quickly the gap emerges — as a function of model size, dataset size, and KL penalty — remains only partially understood.

Reward model ensembles partially mitigate overoptimization: train $M$ independent reward models and optimize against their minimum (pessimistic ensemble), average, or a penalized version that accounts for ensemble disagreement. The minimum-of-ensemble approach has the same structure as TD3's clipped double Q-learning (Week 8) and CQL's conservative Q-function (Week 11) — all are applications of pessimism under uncertainty.

Distributional mismatch in preference data#

The RM is trained on preference pairs from the SFT model's distribution. After PPOProximal Policy Optimisation fine-tuning, the policy's distribution shifts, potentially producing responses that fall outside the RM's training distribution. The RM's predictions become less reliable precisely in the region the policy most wants to exploit. This is the exact offline RLReinforcement Learning distributional shift problem (Week 11) applied in the RLHFReinforcement Learning from Human Feedback context.

One mitigation is iterative RLHFReinforcement Learning from Human Feedback: alternate between (a) collecting new preference data from the current $\pi_\theta$ and (b) updating the reward model on the expanded dataset. Each iteration keeps the RM's training distribution close to the current policy's output distribution, reducing distributional mismatch.

The alignment tax#

RLHFReinforcement Learning from Human Feedback often degrades performance on tasks that were not in the preference dataset. A model fine-tuned for conversational helpfulness may show reduced performance on mathematical reasoning, code generation, or factual recall benchmarks. This tradeoff is the alignment tax: alignment-focused RLHFReinforcement Learning from Human Feedback specializes the model toward preference-annotated behaviors at the cost of general capability.

The alignment tax is partially a consequence of the SFT stage: fine-tuning on a narrow curated dataset risks forgetting the breadth of pretraining knowledge. Approaches including PPOProximal Policy Optimisation with a cross-entropy auxiliary loss (preventing forgetting), careful learning rate scheduling, and mixing SFT and RLReinforcement Learning gradient updates have reduced but not eliminated the alignment tax in practice.

Variants and extensions#

Constitutional AI (CAI) (Bai et al., 2022) replaces human preference annotators with the model itself: the model critiques its own responses against a set of principles (the "constitution") and generates revised responses; a reward model is then trained on these model-generated preference pairs rather than human-annotated ones. This cuts the cost of preference data collection and enables alignment at scales where human annotation is impractical — but at the price of replacing the question "what do humans prefer?" with "what does our constitution imply?" — shifting rather than solving the preference specification problem.

RLAIF (RLReinforcement Learning from AI Feedback) generalizes CAI: the evaluator model can be a separate, stronger LLMLarge Language Model. Preferences are generated by the evaluator rather than humans, then used to train the reward model for the policy. RLAIF can be applied iteratively: the policy improves, the evaluator generates harder preference pairs, the RM improves, the policy improves further. Bai et al. (2022) reported human-competitive alignment at reduced annotation cost in this loop. Lee et al. (2023) compared RLAIFReinforcement Learning from AI Feedback with RLHFReinforcement Learning from Human Feedback directly and found the two reach comparable performance on summarization and on helpful-and-harmless dialogue — not on reasoning benchmarks. Nor is DeepSeek-R1 an RLAIFReinforcement Learning from AI Feedback variant: its reasoning training uses GRPOGroup Relative Policy Optimisation with rule-based verifiable rewards for mathematics and code, supplemented by a preference reward model for general language (DeepSeek-AI, 2025).

Browser lab: KL-penalized reward hacking (RLHF toy)#

Proxy reward $r_\phi$ peaks on a hack string that humans dislike. Policy maximizes $r_\phi - \beta\,\mathrm{KL}(\pi\|\pi_{\mathrm{ref}})$. Small $\beta$ → reward hacking; large $\beta$ stays near the reference.

import numpy as np

# Three completions: good, mediocre, hack
labels = ["helpful", "ok", "hack"]
# True human preference scores (unknown to training)
r_star = np.array([2.0, 0.5, -1.0])
# Proxy RM: hack looks best
r_phi = np.array([1.2, 0.4, 3.5])
# Reference policy (SFT): mostly helpful
pi_ref = np.array([0.70, 0.25, 0.05])

def kl(p, q):
    p = np.clip(p, 1e-12, 1)
    q = np.clip(q, 1e-12, 1)
    return float(np.sum(p * np.log(p / q)))

def optimize(beta, steps=200, lr=0.5):
    # logits for categorical policy
    logits = np.log(pi_ref + 1e-12).copy()
    for _ in range(steps):
        z = logits - logits.max()
        pi = np.exp(z); pi /= pi.sum()
        # objective J = E_pi[r_phi] - beta KL(pi||ref)
        # grad via score: (r_phi - beta (log pi - log ref + 1)) but use softmax path
        adv = r_phi - beta * (np.log(pi + 1e-12) - np.log(pi_ref + 1e-12))
        # policy gradient baseline
        adv = adv - adv.mean()
        logits += lr * adv * pi  # soft update toward high-adv actions
    z = logits - logits.max()
    pi = np.exp(z); pi /= pi.sum()
    return pi

print(f"{'β':>6}  {'π(help)':>8}  {'π(hack)':>8}  {'E[r_φ]':>8}  {'E[r*]':>8}  {'KL':>8}")
for beta in [0.0, 0.05, 0.2, 1.0, 4.0]:
    pi = optimize(beta)
    print(f"{beta:6.2f}  {pi[0]:8.3f}  {pi[2]:8.3f}  {pi @ r_phi:8.3f}  {pi @ r_star:8.3f}  {kl(pi, pi_ref):8.3f}")

print("Notice: without KL the policy mass moves to the RM hack; β anchors π near SFT and preserves true quality.")

What to try: make the RM less wrong (r_phi hack = 1.5) — less need for large $\beta$.

Key takeaways#

The RLHFReinforcement Learning from Human Feedback pipeline maps the alignment problem onto standard RLReinforcement Learning components. SFT shapes the base model into a format suitable for preference optimization and defines the reference distribution. Reward modeling translates pairwise human preferences into a scalar via the Bradley-Terry model and maximum likelihood estimation — avoiding the noise and inconsistency of absolute human scoring. KL-regularized PPOProximal Policy Optimisation optimizes the policy against the RM while constraining it to the SFT distribution, connecting RLHFReinforcement Learning from Human Feedback directly to offline RLReinforcement Learning behavior regularization. The closed-form KL-regularized optimal policy — a Boltzmann distribution over the reference model weighted by reward — is the theoretical centerpiece that links RLHFReinforcement Learning from Human Feedback to maximum entropy RLReinforcement Learning and enables the DPODirect Preference Optimization derivation. Overoptimization, distributional mismatch, and the alignment tax are the principal failure modes of vanilla RLHFReinforcement Learning from Human Feedback, each with mitigation strategies rooted in the RLReinforcement Learning theory developed throughout the course.

Conceptual questions#

1. The Bradley-Terry model assumes $P(y^w \succ y^l) = \sigma(r(x, y^w) - r(x, y^l))$. This model has an identifiability property: adding a constant to all rewards leaves all preference probabilities unchanged. Explain why this means that RLHFReinforcement Learning from Human Feedback cannot learn the absolute scale of the reward function, only the relative ordering. Does this matter for policy optimization? What additional assumption is required to compare the reward of responses to different prompts?

2. The KL-regularized RLHFReinforcement Learning from Human Feedback objective has the closed-form optimal policy $\pi^*(y|x) \propto \pi_{\text{ref}}(y|x) \exp(r(x,y)/\beta)$. Show what happens to this policy as $\beta \to 0$ and $\beta \to \infty$. For a fixed RM $r_\phi$, describe the qualitative behavior of the optimized policy at each extreme. Why does large $\beta$ reduce overoptimization, and what is the cost?

3. The PPOProximal Policy Optimisation implementation of RLHFReinforcement Learning from Human Feedback requires four networks (policy, critic, reference model, reward model). For a 7B parameter model, estimate the minimum GPU memory required if all four are loaded simultaneously in FP16. Then explain why this memory constraint motivates approaches that eliminate the reward model (DPODirect Preference Optimization) or the separate critic (GRPOGroup Relative Policy Optimisation). What architectural compromise does each method make?

4. Overoptimization experiments (Gao et al., 2023) use a gold reward model — a separate, held-out RM trained on additional human data — to evaluate whether the proxy RM score correlates with true preference as KL divergence from $\pi_{\text{SFT}}$ increases. The proxy RM score increases monotonically while the gold RM score peaks and then decreases. Explain this divergence in terms of the distributional shift between the RM's training distribution and the policy's output distribution during PPOProximal Policy Optimisation. What does the peak of the gold RM curve represent, and how would you detect this peak during training without access to a gold RM?

5. Constitutional AI uses model-generated preference pairs rather than human annotations. Identify two ways this approach could fail to produce a well-aligned model even if the constitution is well-specified: one failure mode related to the quality of the evaluator model, and one related to the diversity of the generated preference pairs. Propose a modification to the CAI pipeline that addresses each failure mode.

Knowledge Check#

Check the RLHF pipeline and KL-regularized alignment.

Looking ahead#

The RLHFReinforcement Learning from Human Feedback pipeline is powerful but expensive: four networks, human annotation, and a PPOProximal Policy Optimisation training loop that requires careful hyperparameter tuning.

Week 13: Direct Preference Optimization and GRPOGroup Relative Policy Optimisation. We derive DPODirect Preference Optimization's reparameterization of the reward model in terms of the optimal policy — eliminating the reward model entirely — and study GRPOGroup Relative Policy Optimisation's removal of the critic, tracing what each simplification gives up and what it gains.

Further reading#

• Christiano, P. F., et al. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS. (The conceptual origin of modern RLHFReinforcement Learning from Human Feedback).
• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS. (InstructGPT / OpenAI's RLHFReinforcement Learning from Human Feedback paper).
• Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv. (Anthropic's RLAIF framework).
• Gao, L., et al. (2023). Scaling Laws for Reward Model Overoptimization. ICML. (Empirical study of Goodhart's law in RLHFReinforcement Learning from Human Feedback).
• Stiennon, N., et al. (2020). Learning to summarize from human feedback. NeurIPS. (The summarization result that validated the preference-modeling pipeline).
• Lee, H., Phatale, S., Mansoor, H., et al. (2023). RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv. (Direct comparison: AI feedback matches human feedback on summarization and helpful/harmless dialogue).
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv. (GRPOGroup Relative Policy Optimisation with rule-based verifiable rewards plus a preference reward model — not an RLAIFReinforcement Learning from AI Feedback variant).