Reinforcement Learning: Agents, Rewards, and MDPs

Reinforcement learning is about a learner that acts in a world and learns from what comes back. That one sentence hides the whole subject: what does the learner see, what can it do, and what does it get back?

Acting to do well later#

Picture a robot arm learning to stack blocks. It cannot read a manual, and nobody labels its camera images. Instead it moves, watches what happens, and adjusts. Over many attempts it gets better at the task.

This is the loop that defines reinforcement learning:

1. The agent observes a state — a summary of the situation, such as the arm's joint angles.
2. The agent chooses an action — such as "close the gripper".
3. The world moves to a next state and hands back a reward — a single number, such as $+1$ for a block landing on a block.

Then it repeats. The agent's choices change what it sees later, which changes what it can learn. That feedback loop is what separates reinforcement learning from the other kinds of machine learning:

• In supervised learning, the data is fixed. The model cannot change what it is shown next; it only fits what it was given.
• In reinforcement learning, the agent's own behaviour decides which situations it will face. A timid agent never discovers what happens when it tries something new.

That circularity — the learner creating its own data — is the source of both the power and the difficulty of the field. Everything else in reinforcement learning is bookkeeping for this loop.

The reward: the only signal the agent gets#

At each step the agent receives a reward: one number, good or bad. It is the only feedback it gets. There is no "correct action" label, and no explanation of what it did wrong.

The founding claim of the field is the reward hypothesis:

“

Every goal and purpose can be described as maximising the expected total reward.

A robot that should stack blocks, a system that should recommend videos a person will enjoy, a language model that should produce answers people approve of — each one is written down as a number, and the agent is asked to make that number's running total as large as possible.

This is a claim, not a law of nature. The hard part is not maximising reward; it is writing down a reward that means what you actually want. Two failure modes show up constantly:

• Sparse reward. The number only appears at the very end. A robot that gets $+1$ only when the tower is finished has almost no signal for the hundreds of actions in between.
• Reward hacking. The agent maximises the letter of the reward and violates its spirit. An agent rewarded for how fast its joints move may learn to fall forward rather than walk. An agent rewarded for a proxy of quality — approval, clicks, time on page — will optimise the proxy. The agent is not being malicious; it is doing exactly what it was told, which is precisely the problem.

The world as a board: states and actions#

To say anything precise, we need names for the pieces.

• A state is a snapshot of everything that matters for the decision. In a game of checkers it is the board; in a robot it is the joint angles, velocities, and whatever else the controller needs.
• An action is one of the moves the agent can make from a state.

The pair is enough to describe a decision: given this state, choose an action. The collection of all states is written $\mathcal{S}$, and the collection of all actions is written $\mathcal{A}$.

A tiny example makes this concrete: a $4 \times 4$ grid where a robot starts in one corner and the goal is the opposite corner. The state is the robot's cell — 16 possibilities. The actions are the four moves. The reward is $+1$ on reaching the goal and $0$ elsewhere.

The Markov property: the state has to be enough#

Here is the assumption that makes the whole framework work, stated in plain words:

“

The future depends on the past only through the current state.

If two different histories lead to the same state, then from that state on, they are indistinguishable — everything that mattered has been folded into the state itself. This is the Markov property.

It is a compression claim, and like every compression it can throw away something that mattered. Consider a robot that observes only its joint angles, not how fast the joints are moving. Two robots can report identical angles while moving in opposite directions, and they will go to different places next. The observation is not enough: the same reported state leads to different futures. The Markov property fails.

The fix is called state engineering, and it is normal engineering work, not a mathematical trick:

• Add the missing variable. Give the robot joint velocities as well as angles.
• Stack recent observations. Show the last few camera frames so that motion can be inferred from the differences.
• Learn a summary. Let a network maintain an internal memory of the history and treat that memory as the state.

Most real sensors do not hand you a sufficient state, so almost every practical system begins with this step. When a correct implementation still fails to converge, a non-sufficient state is one of the first things to suspect.

The MDP: five ingredients#

A Markov Decision Process, or MDP, is the standard way to write down a reinforcement-learning problem. It has five ingredients:

Written compactly, an MDP is the tuple

Every term here answers a question a beginner can ask out loud: Where can I be? What can I do? What happens if I do it? What do I get? How much do I care about later? If you can answer those five questions about a problem, you have written it as an MDP.

The transition $P$ is where uncertainty lives. The agent picks an action, but the world decides the outcome — the robot attempts a step and slips, the market moves, the user clicks or does not. The agent's job is to act sensibly even though it cannot control this.

Policies: how the agent chooses#

The agent's behaviour is called a policy, written $\pi$. A policy is a rule that says which action to take in each state:

Two flavours:

• A deterministic policy always makes the same choice in a given state. Good for deployment, where you want predictable behaviour.
• A stochastic policy sometimes tries something else. Good for learning, because an agent that never deviates never discovers that a different action would have been better.

That last point deserves emphasis, because it is the seed of the whole next lesson. Learning requires trying things. If the agent only ever takes the action it currently believes is best, it collects no evidence about the alternatives, and its beliefs can freeze after a single unlucky guess.

Browser lab: a tiny world that answers back#

Run the cell and watch three episodes in a five-position world. The policy never changes — only the world's randomness does.

import numpy as np

rng = np.random.default_rng(0)

# A tiny world: five positions in a line. Position 4 is the goal.
GOAL = 4
SLIP_CHANCE = 0.2  # the world is noisy: 20% of "right" attempts slip left


def policy(position):
    """The agent: a rule from states to actions."""
    return "right"


def env_step(position, action, rng):
    """The environment: state and action in, next state and reward out."""
    slipped = rng.random() < SLIP_CHANCE
    if action == "right" and not slipped:
        next_position = min(position + 1, GOAL)
    else:
        next_position = max(position - 1, 0)
    reward = 1.0 if next_position == GOAL else 0.0
    return next_position, reward


for episode in range(3):
    position = 0
    log = []
    for _ in range(8):
        action = policy(position)                          # agent chooses
        next_position, reward = env_step(position, action, rng)  # world answers
        log.append(f"{position} -{action}-> {next_position} (r={reward:+.1f})")
        position = next_position
        if position == GOAL:
            break
    print(f"episode {episode + 1}: " + " | ".join(log))

print("Notice: the policy is fixed, yet the episodes differ — the same action can have different outcomes because the world is random.")

What to look for. The agent's rule is identical in every episode, so all the variation comes from the environment. The state (the current position) is all the agent needs to choose the next action — this tiny world is Markov by construction. And the reward arrives one step after the action that caused it, which is why the next step is about adding those delayed rewards up.

Key takeaways#

• Reinforcement learning is a loop: state, action, reward, next state, repeat — and the agent's behaviour shapes the data it will learn from.
• Reward is the only feedback. The reward hypothesis says any goal can be written as maximising expected total reward; writing that number correctly is where most failures begin.
• An MDP is five ingredients: states, actions, transition probabilities, rewards, and a discount factor.
• The Markov property asks the state to be a sufficient summary of the past. When observations leave something out — like velocity — the assumption breaks, and the fix is state engineering.
• Policies can be deterministic or stochastic, and stochasticity is what lets an agent learn about actions it is not currently choosing.

Knowledge Check#

Check the building blocks before moving to the arithmetic of value.

Next step#

You now have the world described: states, actions, rewards, and a policy. The next step adds the arithmetic — how the agent counts up rewards over time, what "value" means, and the one equation that turns an infinite sum into a single step.

Next: Returns, Value, and the Bellman Equation