Week 11: Offline Reinforcement Learning

Learning without environment interaction#

Every algorithm studied so far has assumed the agent can interact with the environment during training. That assumption fails in three practically critical settings: when interaction is dangerous (surgical robotics, clinical decision support, autonomous driving), expensive (physical robot hardware, real-time industrial systems), or ethically constrained (deploying exploratory policies on real users). In all three, the agent must learn purely from a fixed dataset collected in the past.

Offline reinforcement learning (also called batch RLReinforcement Learning) studies exactly this setting: given a static dataset $\mathcal{D}$ of transitions, learn the best possible policy without any further environment interaction. The goal is the same as standard RLReinforcement Learning — maximize expected return — but the information constraint is different.

The challenge is not a data-engineering problem. Standard off-policy algorithms (DQNDeep Q-Network, SACSoft Actor-Critic, DDPGDeep Deterministic Policy Gradient) already learn from replay buffers of past experience, yet applied to a fixed offline dataset with no additional collection they fail catastrophically. The cause is a structural property of the Bellman backup and function approximation that does not appear online: distributional shift combined with overestimated Q-values.

The offline RLReinforcement Learning setting#

The agent is given a fixed dataset $\mathcal{D} = \{(s_i, a_i, r_i, s'_i)\}_{i=1}^N$ collected by a behavior policy $\pi_\beta$ — which may be a prior agent, a scripted policy, a human demonstrator, or a mixture. The behavior policy is typically unknown.

The agent must find:

using only transitions in $\mathcal{D}$. It cannot query the environment for new transitions. This is a harder problem than off-policy RLReinforcement Learning, which also learns from a replay buffer: online, the agent still periodically collects fresh data to correct its errors. Offline RLReinforcement Learning has no such correction mechanism.

Dataset quality spectrum#

The difficulty of offline RLReinforcement Learning depends heavily on the dataset:

• Expert data: all transitions come from a near-optimal policy. Behavioral cloning (supervised imitation) already works well here, and offline RLReinforcement Learning adds marginal benefit.
• Scripted or random data: transitions from heuristic or random policies. Value function learning is possible but stitching across different behavior modes is required.
• Mixed data: a combination of expert, random, and possibly suboptimal transitions. This is the most realistic and most difficult setting — the offline RLReinforcement Learning algorithm must identify and improve upon the best parts of the dataset.

The dataset quality matters because the learned policy can only be as good as the best trajectory that can be stitched together from individual transitions in $\mathcal{D}$. Offline RLReinforcement Learning's unique promise — and its central difficulty — is that it can potentially produce a policy better than any individual trajectory in the dataset by combining good segments from different trajectories.

Why standard off-policy RLReinforcement Learning fails offline#

The distributional shift problem#

A standard Q-learning update selects actions by maximizing the Q-function:

The Q-network $Q_\theta$ is trained only on state-action pairs $(s, a)$ present in $\mathcal{D}$. For out-of-distribution (OOD) actions — actions the behavior policy never took in state $s$ — the network's outputs are unconstrained by any training signal. They are whatever the network extrapolates from nearby training points, and that extrapolation can be wildly incorrect.

Because the policy takes the argmax over all actions (including OOD ones), it actively selects OOD actions whenever the network happens to assign them a falsely high Q-value. This is the distributional shift problem: the policy induces a state-action distribution that differs systematically from the dataset distribution $d^{\pi_\beta}(s, a)$.

Bootstrapping error amplifies the problem#

Distributional shift alone would be manageable if the Q-network's errors on OOD actions were random. The deeper problem is that Bellman backups propagate errors. Consider:

If $Q_\theta(s', a')$ is overestimated for some OOD $a'$, this overestimate is used as the target for updating $Q_\theta(s, a)$. On the next backup, this corrupted estimate of $Q_\theta(s, a)$ propagates to states that transition to $s$. Overestimation propagates backward through the value function in the direction of the learned policy's trajectory — precisely the trajectory that visits the most overestimated actions. The result is an explosive feedback loop. $Q_\theta$ overestimates value for some OOD action $a'$ in state $s'$; the policy updates to prefer $a'$; Bellman targets flowing backward from $s'$ become inflated and spread to every state that leads to $s'$; the policy then exploits the now-inflated values throughout the state space. The cycle is absent online, where interaction reveals the true value of $a'$ and corrects the overestimate. Offline it destroys the value function: standard SACSoft Actor-Critic applied to a fixed dataset achieves near-zero return on tasks where behavioral cloning succeeds (Kumar et al., 2020).

Solution 1: Behavior regularization#

The most direct fix is to force the learned policy to remain close to the behavior policy $\pi_\beta$, preventing it from selecting OOD actions the Q-network cannot reliably evaluate.

Policy-constrained objective#

Modify the policy improvement step with a divergence penalty:

where $D$ is a divergence (KL divergence, MMD, or support constraint). The penalty discourages the policy from placing mass on actions not represented in $\pi_\beta$.

BEAR (Kumar et al., 2019) uses Maximum Mean Discrepancy (MMD), which can be estimated without explicitly modeling $\pi_\beta$, only requiring samples from it:

BEAR's insight is that MMD provides a gradient signal for staying in-distribution without a density model of $\pi_\beta$ — samples from it suffice. BEAR matched behavioral cloning on expert data while still improving beyond the behavior policy on mixed datasets, but its dependence on kernel choice and sample count for a reliable MMD estimate limited adoption.

Advantage-Weighted actor-critic (AWAC) takes a simpler approach: update the policy only on actions from the dataset, weighted by their estimated advantage:

This is a supervised update (no off-policy sampling needed) that upweights actions with positive advantage under the current policy. Actions with negative advantage (worse than the policy average) contribute near-zero gradient, so the update is driven by the best subset of the dataset. On D4RL benchmarks AWAC plateaued below the value-regularization methods that followed, which motivated the more aggressive CQL and IQL.

The ceiling of behavior regularization#

Behavior regularization is fundamentally limited to policies close to $\pi_\beta$. If the behavior policy is suboptimal — the usual case in real datasets — the learned policy inherits that suboptimality, and no amount of tuning lifts the ceiling. That bound motivates a different lever: rather than constraining the policy, change the value function so it stops overestimating OOD actions in the first place.

Solution 2: Conservative Q-Learning (CQL)#

Rather than constraining the policy, Conservative Q-Learning (CQL; Kumar et al., 2020) addresses the problem at the level of the value function itself. The key insight is that if we can ensure the Q-function does not overestimate OOD actions, the policy's argmax will remain in-distribution without any explicit policy constraint.

The CQL objective#

CQL modifies the standard Bellman loss to add a pessimism term that pushes down Q-values on OOD actions and pushes up Q-values on dataset actions:

The middle term $\log\sum_a \exp Q_\theta(s,a)$ is the log-sum-exp (soft maximum) over all actions — a differentiable approximation to $\max_a Q_\theta(s,a)$. Minimizing it pushes down the Q-value of whichever actions the network currently thinks are best, disproportionately penalizing overestimated OOD actions. The final term $\mathbb{E}_{a \sim \mathcal{D}}[Q_\theta(s,a)]$ pushes up Q-values for actions in the dataset, preserving the Bellman signal on observed transitions.

CQL's central bet is that pessimism — deliberately underestimating unseen actions — is a feature, not a bug: it prevents the catastrophic divergence standard off-policy methods run into. The bet has a known price, though. On datasets that are already near-optimal, the lower bound also suppresses genuinely good in-distribution actions, so CQL leaves performance on the table relative to less conservative methods.

Theoretical guarantee#

Under mild regularity conditions, CQL guarantees that the learned Q-function is a lower bound on the true value function:

This conservative (pessimistic) bias is intentional. An agent that underestimates the value of OOD actions will avoid them — exactly the desired behavior. The tradeoff is that CQL may also underestimate the value of in-distribution actions, leading to a policy that is overly conservative about deviations from $\pi_\beta$ even when they would be beneficial. The penalty weight $\alpha$ controls this tradeoff: large $\alpha$ is maximally conservative; small $\alpha$ approaches standard Q-learning.

Connection to offline-to-online transfer#

CQL-initialized policies can be fine-tuned online efficiently: the conservative Q-function provides a stable initialization that avoids early-training instability, and online interaction corrects the deliberate underestimation bias. Nair et al. (2020) built AWAC around exactly this premise — offline pre-training followed by online fine-tuning — and the pattern is the dominant deployment pipeline in robotics.

Implicit Q-Learning (IQL)#

A limitation of CQL and behavior regularization approaches is that policy improvement still requires querying the Q-function on actions outside the dataset. Implicit Q-Learning (IQL; Kostrikov et al., 2021) avoids this entirely by reformulating the Bellman backup to never evaluate the Q-function outside the dataset.

The key idea#

Standard Q-learning bootstraps with $\max_{a'} Q(s', a')$, which requires querying Q at the optimal action, which may be OOD. IQL replaces this maximum with an expectile regression over dataset actions:

where $L_\tau^2(u) = |\tau - \mathbf{1}_{u < 0}| \cdot u^2$ is the asymmetric $L_2$ loss. At $\tau = 0.5$ this is ordinary MSE (fitting the mean). At $\tau \to 1$ it fits the maximum Q-value achievable by in-distribution actions. The Q-function is then updated via:

Finally, the policy is extracted by advantage-weighted regression on dataset actions:

Where AWAC constrains the policy and CQL penalizes the value function, IQL attacks the problem at a third point: it rewrites the Bellman backup so OOD actions are never evaluated. Every training step evaluates $Q$ and $V$ only on $(s,a)$ pairs from $\mathcal{D}$, eliminating the OOD-action sampling that drives CQL's estimator variance.

With $\tau = 0.7$–$0.9$, IQL matches or exceeds CQL on D4RL benchmarks and leaves only two hyperparameters to tune, $\tau$ and $\beta$ (Kostrikov et al., 2021). Because the final policy is extracted by the same advantage-weighted supervised loss AWAC uses, the recipe carries over to diffusion policies and other expressive policy classes unchanged.

When IQL applies and when it does not#

IQL is the right tool where behavioral cloning plus mild value improvement is enough — in robotics, where the behavior policy is usually competent but suboptimal. It struggles where the best policy must stitch together disparate trajectory segments: because IQL never evaluates high-value OOD actions, it cannot cross the advantage gap between trajectories. Tasks that demand large improvement beyond the behavior policy are therefore CQL's territory.

Off-policy evaluation#

If the agent cannot interact with the environment during training, it also cannot measure its policy's performance directly. Deploying a poorly performing policy on real hardware or real users is costly. Off-policy evaluation (OPE) estimates the expected return of a target policy $\pi_e$ using only the offline dataset collected by $\pi_\beta$.

Direct method (DM)#

Fit a Q-function $\hat{Q}^{\pi_e}$ using the offline dataset and evaluate it at the initial state distribution:

Simple but biased: all errors in $\hat{Q}$ propagate directly into the estimate.

Importance sampling (IS)#

Reweight trajectory returns by the density ratio between $\pi_e$ and $\pi_\beta$:

IS is unbiased but its variance grows exponentially with trajectory length $T$: the product of $T$ per-step importance weights has variance $\prod_t \mathbb{E}[\rho_t^2] - 1$, which explodes unless $\pi_e \approx \pi_\beta$. For long-horizon tasks IS estimates are effectively unusable.

Doubly robust (DR) estimator#

DR combines the direct method as a variance-reduction baseline with IS for residual correction:

The DR estimator is doubly robust in a precise sense: it is consistent (converges to the true value) if either the importance weights are correct (an accurate $\pi_\beta$ model) or the value model $\hat{Q}$ is correct — not both. In practice DR typically dominates DM and IS whenever one of the two components is moderately well-fit to the data; Kennedy (2023) develops the double-robustness theory and its implications for offline evaluation.

OPE in practice#

In offline deployment OPE is a sanity check, not a definitive metric. Standard practice pairs DR with conservative importance-weight truncation (capping $\rho_t$) to trade bias against variance, then validates on a small batch of real-world data or simulation rollouts when available — no offline estimator substitutes for interaction.

GenAI context: RLHFReinforcement Learning from Human Feedback as offline RLReinforcement Learning#

Offline RLReinforcement Learning is not a niche robotics topic — it is the structural setting of modern LLMLarge Language Model alignment.

In the RLHFReinforcement Learning from Human Feedback pipeline: a fixed dataset of human preference labels is collected once; a reward model is trained on this static data; and the LLMLarge Language Model policy is optimized against the reward model without further human annotation in the training loop. This is offline RLReinforcement Learning on a fixed preference dataset, with the reward model playing the role of $\hat{R}$.

The distributional shift problem appears directly: if PPOProximal Policy Optimisation pushes the LLMLarge Language Model policy to maximize the reward model, it will find OOD text sequences — responses outside the distribution of the preference dataset — that score highly under the reward model but are not actually preferred by humans. This is exactly extrapolation error on OOD actions. The reward model, trained only on prompts and responses from the SFT model, has no reliable predictions for text far from that distribution.

The standard fix is the KL penalty against the SFT reference model:

This is behavior regularization with $\pi_\beta = \pi_{\text{SFT}}$: it prevents the optimized policy from straying into regions where the reward model cannot be trusted. The complete theoretical analysis of this connection — including the closed-form solution of the KL-regularized RLHFReinforcement Learning from Human Feedback objective — is the starting point of the DPODirect Preference Optimization derivation studied in Week 13.

Browser lab: offline Q overestimation + CQL-style penalty#

Two actions. Dataset only covers $a=0$ (behavior). Fitted $Q$ without regularization inflates the unobserved $a=1$; a CQL-like $\log\sum_a e^Q - \mathbb{E}_{a\sim\beta}[Q]$ penalty pulls OOD actions down.

import numpy as np

rng = np.random.default_rng(0)
# True: a=0 safe r~0.5; a=1 risky true mean 0.2 but never in data
# Offline data: only a=0
n = 200
r_data = rng.normal(0.5, 0.15, size=n)
# Tabular Q for one state: Q[0], Q[1]
# Fitted Q-iteration style: minimize (Q[a]-r)^2 on data + alpha * CQL

def fit(alpha_cql, steps=200, lr=0.2):
    Q = np.array([0.0, 0.0])
    for _ in range(steps):
        # Bellman residual on behavior actions (bootstrap with max for "standard" offline)
        # simple: regress Q[0] to mean reward; Q[1] only feels CQL / max backup pressure
        target0 = r_data.mean()
        # standard off-policy backup encourages max_a Q
        y_max = max(Q[0], Q[1])  # dangerous OOD path
        # mix: data fit + soft max backup lure on both
        loss_q0 = (Q[0] - target0) ** 2
        # CQL: logsumexp(Q) - E_beta[Q]
        lse = np.log(np.exp(Q[0]) + np.exp(Q[1]))
        cql = lse - Q[0]  # beta always a=0
        # gradients
        g0 = 2 * (Q[0] - target0) + alpha_cql * (np.exp(Q[0]) / (np.exp(Q[0]) + np.exp(Q[1])) - 1.0)
        g1 = alpha_cql * (np.exp(Q[1]) / (np.exp(Q[0]) + np.exp(Q[1])))
        # also a small "max backup" lure that tries to raise Q1 if unconstrained
        if alpha_cql == 0:
            g1 += 0.15 * (Q[1] - (target0 + 0.3))  # fictitious optimistic OOD TD
        Q[0] -= lr * g0
        Q[1] -= lr * g1
    return Q

print(f"{'α_CQL':>8}  {'Q(a=0)':>8}  {'Q(a=1) OOD':>12}  {'greedy':>8}")
for a in [0.0, 0.5, 1.0, 3.0]:
    Q = fit(a)
    g = int(np.argmax(Q))
    print(f"{a:8.1f}  {Q[0]:8.3f}  {Q[1]:12.3f}  {'a='+str(g):>8}")

print("\nDataset never saw a=1; true E[r|a=1]=0.2. Unregularized OOD Q often wins greedy.")
print("Notice: CQL penalty shrinks OOD Q below behavior; α=0 lets phantom TD invent a=1 value.")

What to try: set the optimistic lure higher (0.3 → 1.0) — without CQL, policy fully flips to OOD.

Key takeaways#

Offline RLReinforcement Learning learns from fixed datasets without environment interaction. Standard off-policy algorithms fail because the Q-function provides unreliable estimates for OOD actions, and Bellman backups propagate these errors into a catastrophic feedback loop. Behavior regularization constrains the policy to stay close to $\pi_\beta$, preventing OOD action selection; AWAC implements this via advantage-weighted supervised regression, enabling direct offline-to-online transfer. CQL addresses the problem at the value function level: a modified Bellman loss explicitly penalizes high Q-values on OOD actions, guaranteeing a conservative lower bound on the true value function. IQL avoids all OOD evaluation by replacing the Bellman max with expectile regression over dataset actions, making it fully in-sample. OPE via doubly robust estimators allows performance evaluation without environment interaction. These methods collectively constitute the machinery underlying RLHFReinforcement Learning from Human Feedback, decision-making from offline datasets in robotics, and any AI system that must learn from historical data without the ability to explore.

Conceptual questions#

1. Standard SACSoft Actor-Critic applied to an offline dataset achieves near-zero performance despite using the same Bellman updates and neural architectures that work in online settings. Trace the failure mechanism step by step: starting from OOD Q-value overestimation in a single state-action pair, show how the error propagates through successive Bellman backups to corrupt the entire value function. Identify the one structural difference between offline and online RLReinforcement Learning that breaks the self-correcting property of online Bellman backups.

2. CQL adds a $\log\sum_a \exp Q_\theta(s,a)$ term to the loss. Show that this term is the log-sum-exp softmax approximation to $\max_a Q_\theta(s,a)$. Why does minimizing this term disproportionately penalize overestimated OOD actions rather than uniformly shrinking all Q-values? How does the tradeoff weight $\alpha$ in CQL relate to the KL penalty $\beta$ in RLHFReinforcement Learning from Human Feedback behavior regularization — are they solving the same underlying problem?

3. IQL fits a value function $V_\psi(s)$ using expectile regression at $\tau \in (0.5, 1)$ on the Q-values of dataset actions. Explain why $\tau = 0.5$ recovers the mean Q-value and $\tau \to 1$ approximates the maximum. In a dataset where the behavior policy is uniformly random over all actions, what does IQL with $\tau = 0.9$ effectively compute, and how does this relate to the optimal value $V^*(s)$?

4. An offline RLReinforcement Learning system for a medical treatment recommendation task is trained on a dataset where $\pi_\beta$ consists of conservative physician decisions that rarely deviate from standard of care. The learned policy learns to recommend aggressive treatments with higher expected return but minimal dataset support. Identify the failure mode using the offline RLReinforcement Learning framework, explain which of the three solution approaches (behavior regularization, CQL, IQL) would best prevent this failure, and describe the tradeoff introduced by applying that approach.

5. The doubly robust OPE estimator is consistent if either the importance weights or the value model is correctly specified, but not necessarily both. Describe a scenario where (a) $\pi_e \approx \pi_\beta$ making IS reliable but the direct method model $\hat{Q}$ is inaccurate, and (b) $\pi_e$ is very different from $\pi_\beta$ making IS unreliable but $\hat{Q}$ is accurate. For case (b), explain what happens to the IS variance as the evaluation horizon $T$ grows, and why DR remains reliable despite high IS variance in this case.

Knowledge Check#

Check offline RL failure modes and fixes.

Implementation exercises#

Exercise 1: Reproduce the offline RL failure#

Train a standard SACSoft Actor-Critic agent online on HalfCheetah-v3 until it reaches a moderate performance level (return ~4000). Save all transitions to create a "medium" offline dataset. Then:

1. Train a new SACSoft Actor-Critic agent offline on this dataset (no environment interaction, just replay buffer sampling). Track the Q-value estimates and policy return over training.
2. You should observe Q-values inflating to 10–100× the true return while actual policy performance collapses to near zero. Plot the divergence between estimated and true Q-values.
3. Add behavior cloning (BC) as a baseline on the same dataset. Observe that BC achieves non-trivial return while offline SACSoft Actor-Critic fails — demonstrating that the failure is specific to value-based methods, not a data quality issue.

Identify the moment in training where the Q-value divergence begins and correlate it with the policy starting to select actions outside the data distribution.

Exercise 2: Implement Conservative Q-Learning (CQL)#

Starting from your offline SACSoft Actor-Critic implementation, add the CQL penalty term:

• For continuous actions, the log-sum-exp term $\log\sum_a \exp Q_\theta(s,a)$ cannot be computed exactly. Use importance sampling: sample $M$ actions from the current policy $\pi_\theta$ and $M$ actions from a uniform distribution over the action space, then compute the soft maximum over the combined set.
• Tune $\alpha \in \{0.1, 1.0, 5.0, 10.0\}$. Report the final policy return after offline training on the medium dataset from Exercise 1.
• Compare with the BC baseline: does CQL exceed BC performance, and at what $\alpha$?

Test on halfcheetah-medium-v2 from D4RL: CQL should achieve a normalized score of ~47 (vs ~42 for BC, ~30 for naive SAC). The d4rl package provides the dataset directly via d4rl.qlearning_dataset(env).

Exercise 3: Compare IQL and CQL on a stitching task#

Construct a custom dataset for PointMaze or a simple navigation task where:

• Half the trajectories go from start to a midpoint and terminate.
• Half go from the midpoint to the goal and terminate.
• No single trajectory goes from start to goal.

Train IQL ($\tau = 0.7, 0.9$) and CQL ($\alpha = 1.0, 5.0$) offline on this dataset. Measure:

• Success rate (start → goal completion) for each method.
• The IQL-CQL performance gap — IQL should struggle (no trajectory covers the full path, and IQL cannot stitch across the gap), while CQL should succeed by identifying the midpoint → goal segment as high-value and crossing into it.

This exercise directly tests the stitching limitation discussed in the IQL section.

Extension prompts#

1. Offline RL with diffusion policies: Replace IQL's Gaussian policy with a diffusion model (as in Diffuser or Decision Diffuser). Train on the same D4RL datasets and compare: (a) performance on heterogeneous data where the behavior policy is a mixture of experts and random actions, and (b) the ability to generate multi-modal action distributions. Does the expressivity of the diffusion policy compensate for IQL's stitching limitation?

2. OPE for model selection without deployment: Implement the doubly robust estimator on the D4RL halfcheetah datasets. Use OPE to rank 5 candidate policies (trained with different offline RL methods on the same dataset) by estimated return. Then evaluate all 5 policies in the real environment. Measure the Spearman rank correlation between OPE-predicted and true returns. At what dataset size does OPE ranking become reliable?

3. RLHF as offline RL — the extrapolation failure: Take a small language model fine-tuned via RLHFReinforcement Learning from Human Feedback (PPOProximal Policy Optimisation with a reward model). Freeze the reward model and generate completions from the optimized policy. Compare the reward model scores with human preference ratings on a held-out set of prompts. Identify text sequences where the reward model score is high but human preference is low — these are the OOD "actions" (text completions) that the reward model overestimates, exactly analogous to the CQL scenario.

Looking ahead#

The machinery built this week — conservative value learning, in-sample backups, behavior regularization — is the substrate for RLHFReinforcement Learning from Human Feedback, the subject of Week 12.

Week 12: Reinforcement Learning from Human Feedback (RLHFReinforcement Learning from Human Feedback). We study how the SFT, reward modeling, and KL-regularized PPOProximal Policy Optimisation pipeline transforms a pre-trained language model into an aligned conversational agent — and examine the limitations of this approach that motivate the preference-optimization methods of Week 13.

Further reading#

• Levine, S., et al. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv. (The definitive overview).
• Kumar, A., et al. (2019). Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. NeurIPS. (BEAR — behavior regularization via MMD constraint).
• Kumar, A., et al. (2020). Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS. (CQL).
• Kostrikov, I., et al. (2021). Offline Reinforcement Learning with Implicit Q-Learning. ICLR. (IQL).
• Nair, A., et al. (2020). Accelerating Online Reinforcement Learning with Offline Datasets. arXiv. (AWAC).
• Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics. (Double robustness theory — the statistical foundation for the DR estimator in OPE).