Week 8: Modern Deep Reinforcement Learning Algorithms

Purpose of this lecture#

Week 7's on-policy methods — TRPOTrust Region Policy Optimisation and PPOProximal Policy Optimisation — use every rollout transition for only a few gradient steps before discarding it. On a physical robot, where each transition costs real hardware time, that waste is disqualifying. This lecture completes the modern deep RLReinforcement Learning toolkit with the off-policy actor-critic family — DDPGDeep Deterministic Policy Gradient, TD3, and SACSoft Actor-Critic — which reuse past transitions from a replay buffer, trading on-policy stability for sample efficiency and dominating physical robotics applications. We develop the family as a progression: DDPGDeep Deterministic Policy Gradient introduces the deterministic policy gradient theorem; TD3 fixes its three core pathologies; SACSoft Actor-Critic replaces the deterministic policy with a maximum-entropy stochastic one and adds automatic entropy tuning. A closing synthesis maps the family onto deployment contexts, in particular the PPOProximal Policy Optimisation vs SACSoft Actor-Critic choice for legged locomotion vs real-robot manipulation.

On-policy recap: TRPOTrust Region Policy Optimisation and PPOProximal Policy Optimisation#

TRPOTrust Region Policy Optimisation and PPOProximal Policy Optimisation were derived in full in Week 7. The key results:

TRPOTrust Region Policy Optimisation constrains each update to a trust region defined by KL divergence:

providing monotone improvement guarantees at the cost of second-order optimization.

PPOProximal Policy Optimisation approximates the trust region with a clipped surrogate:

The $\min$ ensures updates that push $\rho_t$ far from 1 in the direction of improvement are capped; degrading updates are never capped. The full PPOProximal Policy Optimisation loss adds a critic regression term and entropy bonus (see Week 7 for the complete derivation and RLHFReinforcement Learning from Human Feedback mapping).

Why on-policy methods are sample-inefficient: both TRPOTrust Region Policy Optimisation and PPOProximal Policy Optimisation use each rollout transition for only a few gradient steps before throwing it away. Off-policy methods reuse past transitions via a replay buffer, which is the only viable regime when data is scarce — the core motivation for SACSoft Actor-Critic and TD3 on the robots of Course 2. On a Unitree Go1 or manipulation arm, each second of real interaction is expensive, and SACSoft Actor-Critic reaches strong performance within 100K transitions on standard MuJoCo tasks, while Haarnoja et al. (2018, Fig. 1) find it learns considerably faster than PPOProximal Policy Optimisation (as well as DDPGDeep Deterministic Policy Gradient and TD3) — a decisive advantage when every transition must come from real hardware.

The off-policy actor-critic family: overview#

The three off-policy algorithms in this lecture form a clear progression:

Each algorithm inherits from its predecessor and adds targeted fixes, exactly as the DQNDeep Q-Network → Double DQNDeep Q-Network → Dueling → Rainbow progression did in Week 6.

All three share the DQNDeep Q-Network engineering foundation from Week 6: a replay buffer $\mathcal{D}$, target networks updated via Polyak averaging $\theta^- \leftarrow \tau\theta + (1-\tau)\theta^-$, and separate actor and critic networks.

Deterministic policy gradient (DDPGDeep Deterministic Policy Gradient)#

From stochastic to deterministic policies#

Recall the stochastic policy gradient theorem from Week 7:

This requires the policy to be stochastic — the score function $\nabla_\theta \log \pi_\theta(a|s)$ is well-defined only when $\pi_\theta(a|s) > 0$ for all $a$. For continuous action spaces, we may want to learn a deterministic policy $a = \mu_\theta(s)$ that directly outputs an action. The stochastic policy gradient theorem does not apply.

The deterministic policy gradient theorem#

DPG theorem (Silver et al., 2014): For a deterministic policy $a = \mu_\theta(s)$, the gradient of the expected return is:

Derivation. The expected return under the deterministic policy:

where $\rho^\mu(s)$ is the state visitation distribution induced by $\mu_\theta$. Both factors in the integrand depend on $\theta$: $Q^\mu(s, \mu_\theta(s))$ depends on $\theta$ directly through the action $\mu_\theta(s)$, but $\rho^\mu(s)$ also depends on $\theta$, since changing the policy changes which states get visited at all. Applying the product rule to differentiate under the integral sign therefore produces two terms — one from differentiating $Q^\mu(s,\mu_\theta(s))$ with $\rho^\mu(s)$ held fixed, and one from differentiating $\rho^\mu(s)$ itself with $Q^\mu$ held fixed:

The first term is the one that survives and gives the boxed result above. The second term — differentiating the state-visitation distribution through the policy — is exactly the term the stochastic policy gradient theorem also has to dispose of, there via the log-derivative ("score function") trick that turns a derivative of a distribution into an expectation under that distribution. Here Silver et al. (2014) show, under regularity conditions on the MDP, that this second term integrates to zero: this is the nontrivial content of the DPG theorem, not a one-line consequence of the chain rule. The intuition is the same as in the stochastic case — small perturbations to $\theta$ shift where probability mass sits in $\rho^\mu$, but they don't change the total mass (it still integrates to 1), so first-order changes in the visitation distribution alone cannot change $J(\theta)$; only the change in $Q$-value at each visited state can. The full proof (Silver et al., 2014, Theorem 1) works this out formally via the Bellman equation for $\rho^\mu$; we omit it here and simply invoke the result.

Key difference from stochastic PG:

The gradient flows through the Q-function with respect to the action, then through the actor with respect to $\theta$. This requires $Q_\phi(s,a)$ to be differentiable in $a$ — satisfied by neural network critics with continuous action inputs. (That is why DDPGDeep Deterministic Policy Gradient is the default for continuous control in Course 2 robot learning: joint torques are continuous, so $\nabla_a Q$ with respect to joint configurations is well-defined and cheap to evaluate.)

DDPGDeep Deterministic Policy Gradient architecture and exploration#

DDPGDeep Deterministic Policy Gradient (Lillicrap et al., 2015) combines the DPG theorem with DQNDeep Q-Network's engineering:

Initialize actor μ_θ, critic Q_φ, target networks μ_{θ⁻}, Q_{φ⁻}
Initialize replay buffer D

For each step t:
  # Action selection with exploration noise
  aₜ = μ_θ(sₜ) + εₜ,   εₜ ~ N(0, σ²)

  # Environment step, store transition
  Execute aₜ, observe rₜ₊₁, sₜ₊₁
  Store (sₜ, aₜ, rₜ₊₁, sₜ₊₁) in D

  # Sample mini-batch
  Sample {(sⱼ, aⱼ, rⱼ, s'ⱼ)} from D

  # Critic update (DQN-style TD target)
  yⱼ = rⱼ + γ Q_{φ⁻}(s'ⱼ, μ_{θ⁻}(s'ⱼ))
  Minimize Σⱼ (yⱼ - Q_φ(sⱼ, aⱼ))²  over φ

  # Actor update (DPG theorem)
  Maximize Σⱼ Q_φ(sⱼ, μ_θ(sⱼ))  over θ
  (gradient: ∇_θ μ_θ(sⱼ) · ∇_a Q_φ(sⱼ, a)|_{a=μ_θ(sⱼ)})

  # Soft target updates
  θ⁻ ← τθ + (1-τ)θ⁻
  φ⁻ ← τφ + (1-τ)φ⁻

1. Action selection with noise (aₜ = μ_θ(sₜ) + εₜ): the deterministic policy explores only through noise added to the output action.
2. TD target with target networks (yⱼ = rⱼ + γ Q_{φ⁻}(s'ⱼ, μ_{θ⁻}(s'ⱼ))): both target networks are used — the target actor selects the next action, the target critic evaluates it.
3. Critic regression (Minimize Σⱼ (yⱼ - Q_φ(sⱼ, aⱼ))²): minimizes the mean squared Bellman error, exactly as in DQNDeep Q-Network.
4. Actor gradient (Maximize Σⱼ Q_φ(sⱼ, μ_θ(sⱼ))): the actor maximizes the critic's output, the gradient flowing backward from critic to actor via $\nabla_\theta \mu_\theta(s) \cdot \nabla_a Q_\phi(s,a)$.
5. Polyak averaging (θ⁻ ← τθ + (1-τ)θ⁻): soft target updates blend the current weights into the target weights each step — gentler and far more stable than DQN-style hard resets.

Exploration in DDPGDeep Deterministic Policy Gradient is entirely external: the policy $\mu_\theta(s)$ is deterministic, so exploration requires adding noise at action selection time. The original paper uses Ornstein-Uhlenbeck noise (temporally correlated, mean-reverting) to simulate momentum-like exploration in physical systems. In practice, uncorrelated Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$ performs comparably and is simpler. The behavior policy $\beta(a|s) = \delta(a - \mu_\theta(s) - \epsilon)$ differs from the target policy $\pi(a|s) = \delta(a - \mu_\theta(s))$ — DDPGDeep Deterministic Policy Gradient is inherently off-policy, which is why the replay buffer's off-policy data is justified.

DDPGDeep Deterministic Policy Gradient limitations#

Despite its elegance, DDPGDeep Deterministic Policy Gradient has three pathologies in practice:

1. Overestimation bias: the critic target $y = r + \gamma Q_{\phi^-}(s', \mu_{\theta^-}(s'))$ evaluates the target policy's action exactly. Like the single-network DQNDeep Q-Network target, this is susceptible to overestimation — the critic learns to overvalue actions the actor has been encouraged to take, creating a feedback loop.

2. Coupling between actor and critic errors: the actor gradient $\nabla_a Q_\phi$ depends on the quality of the critic. An inaccurate critic provides misleading gradients that update the actor in the wrong direction, degrading the policy, which in turn produces worse training data for the critic. This coupled instability is severe in early training when both networks are randomly initialized.

3. Sensitivity to hyperparameters: the learning rates for the actor and critic must be carefully balanced. If the critic learns too slowly, actor gradients are noisy; if the actor updates too quickly relative to the critic, the policy diverges.

TD3 addresses all three.

Twin delayed deep deterministic policy gradient (TD3)#

TD3 (Fujimoto et al., 2018) introduces three targeted fixes to DDPGDeep Deterministic Policy Gradient's pathologies. Each fix corresponds to a precisely identified failure mode.

Fix 1: clipped double Q-learning#

Maintain two independent critic networks $Q_{\phi_1}$ and $Q_{\phi_2}$. Compute TDTemporal Difference targets using the minimum of the two:

Why the minimum corrects overestimation. Each critic has independent random initialization and independent gradient noise, so their errors are partially decorrelated. The maximum of two estimates is biased upward (Jensen's inequality: $\max(\hat{Q}_1, \hat{Q}_2) \geq Q^*$ in expectation when errors are symmetric). The minimum is biased downward — slightly underestimating is far less harmful than overestimating, because underestimation does not produce the positive feedback loop (overestimated values → actor selects overvalued actions → critic reinforces overvaluation).

This is the continuous-action analog of Double DQNDeep Q-Network from Week 6: decoupling the networks that select and evaluate actions to reduce the max-over-noise bias. (The mechanism matters in Course 2's offline robot learning, where the replay buffer holds demonstrations from suboptimal policies; overestimation would make the robot imitate the worst of them.)

Fix 2: delayed policy updates#

Update the actor less frequently than the critics — typically once every two critic gradient steps:

For each step t:
  Update Q_φ₁ and Q_φ₂  (every step)
  If t mod d == 0:
    Update μ_θ using ∇_a Q_φ₁    (every d steps, d=2 typically)
    Update all target networks

Why delayed updates stabilize training. The actor gradient $\nabla_\theta \mu_\theta(s) \cdot \nabla_a Q_{\phi_1}(s,a)|_{a=\mu_\theta(s)}$ is only meaningful when the critic $Q_{\phi_1}$ is a reliable estimate. In early training, the critic is inaccurate, and updating the actor from a bad critic pushes the policy in the wrong direction. Delaying actor updates gives the critics more time to converge on each policy before the policy is updated.

Fix 3: target policy smoothing#

Add clipped noise to the target policy's action when computing the TDTemporal Difference target:

Why this regularizes the critic. A deterministic policy evaluates the critic at exactly one action per state. If $Q_\phi(s,a)$ develops a sharp peak at $a = \mu_\theta(s)$ (which the actor is incentivized to exploit), the critic will overfit to these exact actions and generalize poorly. Adding noise smooths the target: instead of fitting $Q$ to the value at a single point, the critic must fit the value in a neighborhood of actions around $\mu_{\theta^-}(s')$. The clipping $[-c, c]$ ensures the smoothed action stays near the target policy and does not drift to irrelevant regions of the action space.

TD3 pseudocode summary#

Initialize μ_θ, Q_φ₁, Q_φ₂ and target networks μ_{θ⁻}, Q_{φ₁⁻}, Q_{φ₂⁻}
Initialize replay buffer D

For each step t:
  aₜ = μ_θ(sₜ) + ε,   ε ~ N(0, σ²)
  Store (sₜ, aₜ, rₜ₊₁, sₜ₊₁) in D
  Sample mini-batch from D

  # Smoothed target action
  ã' = μ_{θ⁻}(s') + clip(N(0,σ̃²), -c, c)

  # Critic targets (clipped double Q)
  y = r + γ · min(Q_{φ₁⁻}(s', ã'), Q_{φ₂⁻}(s', ã'))

  # Update both critics
  Minimize (y - Q_φᵢ(s, a))² for i = 1, 2

  # Delayed actor update
  If t mod d == 0:
    Maximize Q_φ₁(s, μ_θ(s))  over θ
    Update target networks via Polyak averaging

1. Target Policy Smoothing (ã' = μ_{θ⁻}(s') + clip(N(0,σ̃²), -c, c)): clipped noise on the target action stops the critic overfitting to the deterministic actor output.
2. Clipped Double Q-Learning (y = r + γ · min(Q_{φ₁⁻}(s', ã'), Q_{φ₂⁻}(s', ã'))): both target critics evaluate the smoothed action and the minimum is taken, removing the overestimation bias endemic to Q-learning.
3. Critic updates (Minimize (y - Q_φᵢ(s, a))² for i = 1, 2): both critics regress to the same conservative target.
4. Delayed Policy Updates (If t mod d == 0): the actor and target networks update only every d critic steps, so the actor optimizes against a settled critic.
5. Actor update (Maximize Q_φ₁(s, μ_θ(s))): maximizes $Q_{\phi_1}$; only one critic is needed to supply the actor's gradient.

Soft actor–critic (SACSoft Actor-Critic)#

SACSoft Actor-Critic (Haarnoja et al., 2018) replaces the deterministic policy with a maximum entropy framework, changing the problem the actor-critic is solving at a fundamental level.

The maximum entropy objective#

Standard RLReinforcement Learning maximizes expected return:

Maximum entropy RLReinforcement Learning augments this with a policy entropy bonus at every timestep:

where $\mathcal{H}(\pi(\cdot|s)) = -\mathbb{E}_{a \sim \pi}[\log \pi(a|s)]$ and $\alpha > 0$ is the temperature parameter controlling the entropy-reward tradeoff. This is not just reward shaping — it changes the definition of optimality.

The soft Bellman equation#

The maximum entropy objective admits a modified Bellman equation. Define the soft Q-function $Q_{\text{soft}}^*(s,a)$ as the solution to:

where the soft value function integrates over both reward and entropy:

The last equality (the log-sum-exp / soft maximum) follows from the closed-form optimal policy:

The optimal maximum entropy policy is a Boltzmann distribution over the Q-function, with temperature $\alpha$. As $\alpha \to 0$, this collapses to a deterministic policy selecting the greedy action. As $\alpha \to \infty$, the policy approaches uniform — all actions are equally likely. At intermediate $\alpha$, the policy concentrates on high-value actions while maintaining uncertainty in lower-value regions.

This is a principled closed-form result, not a heuristic: maximum entropy RLReinforcement Learning has a well-defined optimal policy, and SACSoft Actor-Critic approximates it.

The reparameterization trick#

PPOProximal Policy Optimisation estimates policy gradient via the score function (Week 7):

This is high-variance because $f(a)$ and $\nabla_\theta \log \pi_\theta$ are correlated through different paths. SACSoft Actor-Critic uses the reparameterization trick: instead of sampling $a \sim \pi_\theta(\cdot|s)$ directly, write:

where $f_\theta(s,\xi) = \mu_\theta(s) + \sigma_\theta(s) \odot \xi$ (for diagonal Gaussian, typically with tanh squashing for bounded actions). Then:

The gradient now flows directly through $Q(s,a)$ with respect to $a = f_\theta(s,\xi)$, then through $f_\theta$ with respect to $\theta$. This is lower-variance than the score function estimator because the backpropagation path is deterministic (noise $\xi$ is fixed when differentiating). The reparameterization trick is what enables SACSoft Actor-Critic to use a stochastic policy while maintaining the low-variance gradient of the DPG theorem.

Automatic entropy tuning#

The temperature $\alpha$ controls the exploration-exploitation tradeoff: high $\alpha$ encourages diverse actions; low $\alpha$ focuses the policy. Manually tuning $\alpha$ as a hyperparameter requires task-specific knowledge. SACSoft Actor-Critic automatically adjusts $\alpha$ by formulating it as a constrained optimization: maintain a target entropy $\mathcal{H}_{\text{target}}$ and find the $\alpha$ that achieves it.

Applying dual gradient descent (Lagrangian duality) to the constraint $\mathbb{E}[\mathcal{H}(\pi)] \geq \mathcal{H}_{\text{target}}$:

In practice, $\alpha$ is updated alongside the actor and critic via a separate gradient step:

If the current policy entropy is above $\mathcal{H}_{\text{target}}$, this decreases $\alpha$ (reduce exploration penalty); if below, it increases $\alpha$ (increase exploration). The system self-regulates. A common choice is $\mathcal{H}_{\text{target}} = -\dim(\mathcal{A})$ (negative of action dimension), which works well across continuous control tasks without manual tuning.

This is the same Lagrangian dual ascent used for temperature optimization in Haarnoja et al. (2018); it is analogous to the automatic KL-coefficient tuning in some RLHFReinforcement Learning from Human Feedback implementations.

SACSoft Actor-Critic pseudocode#

Initialize actor π_θ (stochastic: outputs μ_θ(s), σ_θ(s))
Initialize critics Q_φ₁, Q_φ₂ and target networks Q_{φ₁⁻}, Q_{φ₂⁻}
Initialize temperature α (or α-network for auto-tuning)
Initialize replay buffer D

For each step t:
  # Stochastic action via reparameterization
  aₜ = μ_θ(sₜ) + σ_θ(sₜ) ⊙ ξ,   ξ ~ N(0, I)
  Store (sₜ, aₜ, rₜ₊₁, sₜ₊₁) in D

  Sample mini-batch from D
  Sample â' = μ_θ(s') + σ_θ(s') ⊙ ξ',  ξ' ~ N(0, I)

  # Critic targets (soft Bellman, clipped double Q)
  y = r + γ · (min(Q_{φ₁⁻}(s', â'), Q_{φ₂⁻}(s', â')) - α log π_θ(â'|s'))

  # Update both critics
  Minimize (y - Q_φᵢ(s, a))²  for i = 1, 2

  # Actor update (reparameterization trick)
  â = μ_θ(s) + σ_θ(s) ⊙ ξ
  Maximize  min(Q_φ₁(s, â), Q_φ₂(s, â)) - α log π_θ(â|s)  over θ

  # Temperature update (auto-tuning)
  Minimize  α(-log π_θ(â|s) - H_target)  over α

  # Soft target updates
  φᵢ⁻ ← τφᵢ + (1-τ)φᵢ⁻  for i = 1, 2

Note three connections to prior lectures: the clipped double Q target (Week 6: Double DQNDeep Q-Network), the soft Bellman entropy term $-\alpha \log \pi_\theta(\hat{a}'|s')$ in the critic target (the maximum entropy modification), and the reparameterization gradient in the actor update (as opposed to the score function estimator in PPOProximal Policy Optimisation).

Algorithm synthesis#

Expanded comparison table#

PPOProximal Policy Optimisation vs SACSoft Actor-Critic: the robotics decision#

In practice, the choice between PPOProximal Policy Optimisation and SACSoft Actor-Critic depends on the data collection regime:

Choose PPOProximal Policy Optimisation when:

• Fast parallel simulation is available (IsaacLab, MuJoCo with 4096 parallel envs). On-policy inefficiency is irrelevant when you can collect millions of transitions per second in simulation. PPOProximal Policy Optimisation with GAE achieves excellent results in legged locomotion (ANYmal, Unitree, Spot) precisely because massive parallelism compensates for sample inefficiency.
• Reward shaping is complex. On-policy data always reflects the current policy, making reward shaping and curriculum learning easier to reason about. Off-policy replay buffers contain stale data from old policies, which can interact badly with changing reward functions.
• Implementation simplicity matters. PPOProximal Policy Optimisation has fewer moving parts than SACSoft Actor-Critic — no automatic entropy tuning, no reparameterization, no double critics. RSL-RLReinforcement Learning's PPOProximal Policy Optimisation implementation for legged locomotion is 484 lines (rsl_rl/algorithms/ppo.py; Rudin et al., 2022), whereas a full SACSoft Actor-Critic implementation is substantially more complex.

Choose SACSoft Actor-Critic when:

• Real-robot data collection is the bottleneck. SACSoft Actor-Critic's replay buffer reuses every transition, so it needs far fewer interactions than on-policy PPOProximal Policy Optimisation to reach the same performance (Haarnoja et al., 2018). For Franka arm manipulation or other real-hardware setups, this difference is decisive.
• Exploration quality matters more than speed. SACSoft Actor-Critic's entropy-driven exploration produces smoother, more diverse trajectories than $\epsilon$-greedy or additive noise, which is particularly valuable for contact-rich manipulation tasks where the agent must discover narrow-margin contact modes.
• Hyperparameter robustness is needed. Automatic entropy tuning makes SACSoft Actor-Critic largely self-configuring on the temperature hyperparameter, reducing the tuning burden compared to PPOProximal Policy Optimisation's entropy coefficient and GAE $\lambda$.

The hybrid regime: many modern systems train with PPOProximal Policy Optimisation in simulation (fast, stable, handles complex reward shaping), then fine-tune on real hardware with SACSoft Actor-Critic (sample efficient, smooth exploration).

GenAI context: PPOProximal Policy Optimisation for RLHFReinforcement Learning from Human Feedback and SACSoft Actor-Critic for structured generation#

PPOProximal Policy Optimisation in RLHFReinforcement Learning from Human Feedback (recap from Week 7): the language model $\pi_\theta$ is the actor, the reward model $r_\phi$ provides the return signal, and the KL penalty against the SFT reference model prevents reward hacking. PPOProximal Policy Optimisation's on-policy sampling is acceptable in this setting because the language model generates its own rollouts (completions), and on-policy data ensures the distribution of completions matches the current policy — important for RLHFReinforcement Learning from Human Feedback where the reward model may be poorly calibrated outside the SFT distribution.

SACSoft Actor-Critic-inspired ideas in GenAI: maximum entropy RLReinforcement Learning has influenced several GenAI directions:

• Temperature sampling in language model decoding is the inference-time analog of the Boltzmann optimal policy $\pi^*(a|s) \propto \exp(Q/\alpha)$ — high temperature encourages diverse completions; low temperature is greedy.
• Energy-based models for text generation model the distribution over sequences as $p(x) \propto \exp(r(x)/\alpha)$, the same Boltzmann form as the SACSoft Actor-Critic optimal policy.
• DPODirect Preference Optimization and GRPOGroup Relative Policy Optimisation (next lecture) eliminate the critic and reformulate RLHFReinforcement Learning from Human Feedback as a maximum-entropy optimization directly on preference data, removing the need for PPOProximal Policy Optimisation entirely.

Browser lab: twin Q min (TD3) + soft max-ent policy (SAC)#

Two mechanisms from modern continuous-control actors, in 1-D discrete action form:

1. TD3: $\min(Q_1,Q_2)$ cuts overestimation when one critic is optimistic.
2. SAC: $\pi(a)\propto e^{Q/\alpha}$ — temperature $\alpha$ trades return vs entropy.

import numpy as np

rng = np.random.default_rng(0)
# 5 actions, true Q*; two noisy critics (one systematically high)
A = 5
Q_star = np.array([0.5, 1.0, 0.2, 1.4, 0.8])
Q1 = Q_star + rng.normal(0, 0.15, size=A) + 0.35  # optimistic bias
Q2 = Q_star + rng.normal(0, 0.15, size=A)

def greedy(Q):
    return int(np.argmax(Q))

print("True best action:", greedy(Q_star), "Q*=", Q_star[greedy(Q_star)])
print("Q1 greedy:       ", greedy(Q1), " (overest risk)")
print("min(Q1,Q2) greedy:", greedy(np.minimum(Q1, Q2)))
print(f"  max Q1={Q1.max():.2f}  max min={np.minimum(Q1,Q2).max():.2f}  true max={Q_star.max():.2f}")

def soft_policy(Q, alpha):
    z = Q / alpha
    z = z - z.max()
    p = np.exp(z)
    return p / p.sum()

print("\nSAC-style soft policy π(a) ∝ exp(Q*/α):")
print(f"{'α':>6}  {'entropy':>8}  {'E[Q]':>8}  {'p(best)':>8}")
best = greedy(Q_star)
for alpha in [0.1, 0.5, 1.0, 2.0, 5.0]:
    p = soft_policy(Q_star, alpha)
    H = -np.sum(p * np.log(p + 1e-12))
    print(f"{alpha:6.1f}  {H:8.3f}  {p @ Q_star:8.3f}  {p[best]:8.3f}")

print("Notice: twin min prefers safer targets; low α → near-greedy, high α → explore (max-ent).")

What to try: increase Q1 bias to +1.0 — single-critic greedy picks the wrong arm more often; min still tracks $Q^*$.

Key takeaways#

The lecture develops the off-policy actor-critic family as a principled progression. DDPGDeep Deterministic Policy Gradient combines the deterministic policy gradient theorem — gradient flows through $\nabla_a Q$ rather than $\nabla_\theta \log \pi$ — with DQNDeep Q-Network's replay buffer and target networks, but suffers from overestimation, coupled instability, and sensitivity to hyperparameters. TD3 applies three targeted fixes: clipped double Q-learning reduces overestimation via the minimum of two independent critics (continuous-action Double DQNDeep Q-Network); delayed policy updates decouple critic convergence from actor updates; target policy smoothing prevents the critic from overfitting to sharp peaks at deterministic actions. SACSoft Actor-Critic replaces the deterministic policy with the maximum entropy framework, whose optimal policy is provably a Boltzmann distribution over the soft Q-function. The reparameterization trick provides lower-variance gradients than score function estimation. Automatic entropy tuning via dual gradient descent eliminates the temperature hyperparameter.

The PPOProximal Policy Optimisation vs SACSoft Actor-Critic choice maps cleanly onto deployment context: PPOProximal Policy Optimisation with fast parallel simulation for legged locomotion, SACSoft Actor-Critic for real-robot data-scarce settings, with hybrid workflows bridging the two.

Conceptual questions#

1. The deterministic policy gradient theorem gives $\nabla_\theta J(\theta) = \mathbb{E}[\nabla_\theta \mu_\theta(s) \cdot \nabla_a Q^\mu(s,a)|_{a=\mu_\theta(s)}]$. Compare this with the stochastic policy gradient theorem from Week 7. In the stochastic case, where does the gradient come from? In the deterministic case, what must be true about the critic $Q_\phi(s,a)$ for this gradient to be well-defined? Why does DDPGDeep Deterministic Policy Gradient use a Q-critic rather than a V-critic?

2. TD3 uses the minimum of two Q-estimates as the critic target: $y = r + \gamma \min_{i=1,2} Q_{\phi_i^-}(s', \tilde{a}')$. Using the minimum rather than the mean introduces a downward bias. Explain why downward bias is less harmful than upward bias in the actor-critic setting by tracing the feedback loop that upward bias (overestimation) creates between the actor and critic.

3. SACSoft Actor-Critic's optimal policy is $\pi^*(a|s) \propto \exp(Q_{\text{soft}}^*(s,a)/\alpha)$. Show what happens to this policy as $\alpha \to 0$ and as $\alpha \to \infty$. Then explain: if $\alpha$ is too large during training, what behavior do you expect from the agent? If $\alpha$ is too small, what problem arises? How does automatic entropy tuning resolve this without manual intervention?

4. Compare exploration mechanisms across the three off-policy algorithms: DDPGDeep Deterministic Policy Gradient adds external Gaussian noise to the deterministic action; TD3 adds smoothed noise to the target policy (not the behavior policy); SACSoft Actor-Critic's stochastic policy provides intrinsic exploration. For a manipulation task requiring precise contact (narrow-margin task), which exploration mechanism is most appropriate and why? What happens to DDPGDeep Deterministic Policy Gradient-style exploration when the task reward requires very precise actions?

5. You are training a quadruped locomotion policy. You have two setups: (A) IsaacLab with 4096 parallel simulation environments, each running at 100 Hz; (B) a single physical Unitree Go1 robot collecting ~1 Hz data due to safety stops. For each setup, argue which algorithm — PPOProximal Policy Optimisation or SACSoft Actor-Critic — is more appropriate, citing sample efficiency, exploration quality, and implementation complexity. For setup B, what modifications to the standard SACSoft Actor-Critic algorithm would you make to handle the safety constraints of real hardware?

Knowledge Check#

Check the off-policy continuous-control family from this week.

Implementation exercises#

Exercise 1: DDPG core loop#

Implement the DDPG training loop in Python using a deep learning framework of your choice. Start from the pseudocode above. Key implementation details:

• Critic architecture: The critic takes both state and action as input and outputs a scalar Q-value. Concatenate action after a hidden layer (not at the input) for numerical stability.
• Exploration noise: Start with simple Gaussian noise $\epsilon \sim \mathcal{N}(0, 0.1)$ — it performs comparably to Ornstein-Uhlenbeck and is simpler to debug.
• Polyak averaging: Use $\tau = 0.005$ for target network updates. Verify that target_actor.parameters() track actor.parameters() with a small lag.
• Training ratio: One gradient step per environment step. Use a replay buffer of size $10^6$ and batch size 64.

Test on Pendulum-v1: The agent should solve it (average return > −200 over 100 episodes) within 100 episodes.

Exercise 2: Upgrade DDPG to TD3#

Starting from your DDPG implementation, add the three TD3 fixes one at a time, testing after each:

1. Add a second critic network ($Q_{\phi_2}$). Use $\min(Q_1, Q_2)$ for target computation. Track the Q-value gap $\max(Q_1, Q_2) - \min(Q_1, Q_2)$ during training — it should shrink as both critics converge.
2. Delay the actor update to every 2 critic steps ($d = 2$). Plot policy entropy over time — a stable TD3 policy should show less oscillation than DDPG.
3. Add target policy smoothing with $\sigma = 0.2$, $c = 0.5$. Compare critic gradient variance with and without smoothing.

Test on HalfCheetah-v4 (MuJoCo): TD3 should outperform DDPG by approximately 30% in final return (Fujimoto et al., 2018).

Exercise 3: SAC with automatic entropy tuning#

Implement SACSoft Actor-Critic extending your TD3 codebase. Key additions:

• Stochastic actor: Outputs $\mu_\theta(s)$ and $\log \sigma_\theta(s)$ for a diagonal Gaussian. Apply tanh squashing for bounded actions (add log-probability correction for the Jacobian of tanh).
• Reparameterization trick: Sample $a = \tanh(\mu_\theta(s) + \sigma_\theta(s) \odot \xi)$ with $\xi \sim \mathcal{N}(0, I)$. The gradient flows through $Q(s, a)$ via autodiff — not through a score function estimator.
• Soft Bellman target: $y = r + \gamma(\min_i Q_{\phi_i^-}(s', a') - \alpha \log \pi_\theta(a'|s'))$.
• Automatic entropy tuning: Initialize $\log \alpha$ (not $\alpha$) to ensure positivity. Target entropy $\mathcal{H}_{\text{target}} = -\dim(\mathcal{A})$. Update $\alpha$ with a separate optimizer (learning rate $3 \cdot 10^{-4}$).

Test on Ant-v4 (MuJoCo): SAC should achieve approximately 6000 return in 1M steps — 3–5× better sample efficiency than TD3 (Haarnoja et al., 2018b).

Extension prompts#

1. Continuous-action Rainbow: Combine TD3's clipped double Q with SAC's maximum-entropy framework and automatic temperature tuning. How does this hybrid perform on Humanoid-v4 compared to either algorithm alone? What does the entropy curve reveal about the benefit of stochastic exploration in high-dimensional action spaces?

2. Offline RL via CQL: Collect 1M transitions from a partially trained SAC agent. Train a new policy from this fixed dataset using Conservative Q-Learning (CQL), which penalizes overestimated Q-values for out-of-distribution actions. Compare with standard SAC trained on the same offline data. Which shows less extrapolation error?

3. Sim-to-real gap: Train a policy in MuJoCo, deploy it in a different physics engine (e.g., PyBullet or Isaac Sim), and measure the performance gap. Explore domain randomization (varying mass, friction, joint damping ranges) as a mitigation — how much randomization is needed before the policy transfers without fine-tuning?

Further reading#

• Silver, D., et al. (2014). Deterministic Policy Gradient Algorithms. ICML. (DPG Theorem).
• Lillicrap, T. P., et al. (2015). Continuous control with deep reinforcement learning. ICLR. (DDPGDeep Deterministic Policy Gradient).
• Fujimoto, S., et al. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ICML. (TD3).
• Haarnoja, T., et al. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML. (SACSoft Actor-Critic).
• Haarnoja, T., et al. (2018b). Soft Actor-Critic Algorithms and Applications. arXiv. (Automatic entropy tuning and practical SAC implementation details).
• Duan, Y., et al. (2016). Benchmarking Deep Reinforcement Learning for Continuous Control. ICML. (Standardized evaluation benchmarks for DDPG, TRPO, and other continuous-control algorithms).
• Rudin, N., Hoeller, D., Reist, P., & Hutter, M. (2022). Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. CoRL. (The rsl_rl library; the source of the 484-line PPO implementation used for legged locomotion — leggedrobotics/rsl_rl).