Returns, Value, and the Bellman Equation

An agent does not receive one reward — it receives a stream of them, wrapped around a sequence of choices. To compare two policies we need to collapse that stream into a single number. That number is the return, and the equation that computes it recursively is the Bellman equation.

From one reward to a running total#

Suppose an agent has just received reward $r_{t+1}$, then $r_{t+2}$, and so on. Its goal is not to maximise the next reward; it is to maximise the whole stream. So we define the return from time $t$:

Read it aloud: each reward further into the future is worth less, and the discount factor $\gamma$ says how much less. The reward received immediately is counted at full value; the one after that is multiplied by $\gamma$; the one after that by $\gamma^2$; and so on.

The discount factor is a number between 0 and 1. Two extremes show what it means:

• With $\gamma$ close to 0, only the very next reward matters. The agent is short-sighted.
• With $\gamma$ close to 1, rewards far in the future count almost as much as immediate ones. The agent is patient.

Three jobs of the discount factor#

Discounting is often taught as "the agent prefers rewards sooner", but it is doing more work than that.

1. It keeps the total finite. If an agent earns $+1$ at every step of a task that never ends, the undiscounted sum is $1 + 1 + 1 + \cdots$, which grows without bound. Multiplying the $k$-th term by $\gamma^k$ makes the sum of a geometric series with ratio $\gamma$, so it converges to a finite number. Without that, there is no single number to maximise.

2. It encodes time preference. Discounting is how we express that a reward now is worth more than the same reward later — the same idea as interest rates in finance. A smaller $\gamma$ makes the agent care less about a distant payoff.

3. It sets the planning horizon. The weights $\gamma^k$ shrink geometrically, so only a certain number of steps ahead carry real weight. Because $1 + \gamma + \gamma^2 + \cdots = \frac{1}{1-\gamma}$, the effective horizon is about $\frac{1}{1-\gamma}$ steps. At $\gamma = 0.9$ that is ten steps; at $\gamma = 0.99$ it is a hundred. This is a practical dial: how far ahead should the agent plan?

The return, written out#

Two worked numbers make the formula concrete. Take $\gamma = 0.9$ and suppose the agent receives $+1$ at each of five steps:

The five rewards add up to 5, but their discounted total is about 4.10. Now let the rewards continue forever. The sum becomes the geometric series $1 + \gamma + \gamma^2 + \cdots$, which equals

An infinite number of rewards adds up to a finite 10 because each one is discounted more than the last. That closed form is worth remembering: it appears again and again whenever a state keeps earning the same reward forever.

Value: the expected return#

The return is a property of one trajectory. Policies are judged on average, so we take the expected return and call it value.

The state-value function of a policy $\pi$ answers "how good is it to be here, if I keep following $\pi$?":

The action-value function answers "how good is it to be here and take this action first?":

The difference is worth saying in plain words:

• $V^{\pi}(s)$ — how good is this situation?
• $Q^{\pi}(s, a)$ — how good is this situation, if I start with action $a$?

The action value is more informative, because it directly compares the actions available. The state value is a weighted average of them, where the weights are the policy's action probabilities:

If we could compute $Q^{*}$ — the action values of the best policy — the problem would be solved: in each state, pick the action with the largest $Q^{*}$. Everything in reinforcement learning is a way of estimating these numbers without knowing the future.

The Bellman idea: reward now, value later#

Here is the step that turns an infinite problem into a computable one. Write the return as its first term plus the rest:

Then take expectations. The value of a state equals the reward you get now plus the discounted value of where you land:

“

The value of now is the reward now, plus the discounted value of next.

That sentence is the Bellman equation, and it holds exactly. For a fixed policy, averaging over both the policy's action choice and the world's randomness gives the Bellman expectation equation:

The outer sum averages over the action the policy might take; the inner sum averages over the state the world might deliver. The equation is recursive: $V^{\pi}$ appears on both sides. Notice there are no infinite sums left — every term refers only to the value of a next state.

For the best possible policy, each state simply takes the best action instead of averaging:

This is the Bellman optimality equation. The change looks small — a $\max$ where an average used to be — but it is the difference between evaluating a policy and finding the best one. The $\max$ also makes the equation non-linear, which is why optimality is solved by iteration rather than a single linear solve.

Why the recursion settles#

The Bellman right-hand side defines an update: take a guess at $V$, apply the equation, get a better guess, repeat. Why does that not wander forever?

Because multiplying by $\gamma$ shrinks differences. If two guesses are far apart, one sweep of the Bellman update pulls them closer by a factor of $\gamma$ at worst. Squeeze a distance by a factor below 1 and the guesses must converge to a single fixed point — one set of values that reproduces itself under the equation. This is the contraction property, and it is the mathematical guarantee that iterative methods like value iteration terminate at the right answer.

The guarantee is exactly why $\gamma$ must be below 1 for continuing tasks. At $\gamma = 1$ the shrinking factor is 1 — nothing is squeezed, there may be no unique fixed point, and the iteration can cycle. Discounting is not just about patience; it is what makes the problem well-posed.

Browser lab: returns and the Bellman fixed point#

The cell computes a discounted return directly, then finds the same answer by iterating the Bellman equation.

import numpy as np

gamma = 0.9

# 1. The discounted return, computed the long way.
rewards = np.array([1.0, 1.0, 1.0, 1.0, 1.0])
discounts = gamma ** np.arange(len(rewards))
G = np.sum(discounts * rewards)
print(f"5 rewards of +1, discounted at {gamma}: {G:.4f}")

# 2. The same idea for a state that earns +1 forever. Bellman says V = 1 + gamma * V.
V = 0.0
for _ in range(200):
    V = 1.0 + gamma * V
print(f"Bellman fixed point: {V:.4f}  (closed form 1/(1-gamma) = {1 / (1 - gamma):.4f})")

# 3. Watch a guess walk up to the fixed point instead of jumping to it.
V = 0.0
snapshots = []
for sweep in range(20):
    V = 1.0 + gamma * V
    if sweep in (0, 1, 2, 4, 9, 19):
        snapshots.append(f"after {sweep + 1:>2} sweeps: V = {V:.4f}")
print("Approaching the fixed point:")
for line in snapshots:
    print("  " + line)

print("Notice: each sweep shrinks the remaining gap by gamma, so the guesses converge geometrically.")

What to look for.

• The five-step return (about 4.10) is smaller than the infinite-state value (10.0) — the infinite value keeps earning after the fifth step.
• Every sweep multiplies the remaining gap by $\gamma = 0.9$. The starting error is 10, so after ten sweeps it is about 3.5, and after twenty it is about 1.2. That is the contraction at work, and it is why a discount below 1 matters.

Try it. Change gamma to 0.99 and re-run. The closed form jumps to 100, but 200 sweeps are no longer enough to get there — the printed value lands near 86.6, and convergence is visibly slower. A larger effective horizon costs more iterations. Then try gamma = 1.0 with the loop in part 2 and watch V grow without settling.

Key takeaways#

• The return $G_t = \sum_k \gamma^k r_{t+k+1}$ collapses a reward stream into one number by discounting the future geometrically.
• The discount factor does three jobs: keeps the total finite, encodes time preference, and sets an effective planning horizon of about $\frac{1}{1-\gamma}$ steps.
• Value functions are expected returns. $V^{\pi}(s)$ is "how good is this state", $Q^{\pi}(s,a)$ is "how good is this state plus this action".
• The Bellman equation is the recursion $V = \text{reward now} + \gamma \cdot V(\text{next})$. The expectation version evaluates a policy; the max version describes the best one.
• With $\gamma$ below 1 the Bellman update shrinks the error by $\gamma$ each sweep, so iteration converges to a unique fixed point. At $\gamma = 1$ that guarantee disappears.

Knowledge Check#

Check the arithmetic and the idea behind the recursion.

Next step#

Value functions tell an agent how good its current choices are. The next step is about the other half of learning: how an agent gathers the evidence it needs, what it costs to try an uncertain option, and how to measure the reward it gave up along the way.

Next: Exploration, Exploitation, and Regret