Learn AI with Python · Lesson

Policy Gradient Methods: REINFORCE

Policy gradient theorem, REINFORCE algorithm, baseline subtraction, variance reduction.

Policy Gradient Methods: REINFORCE is a free Learn AI with Python lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Value-Based vs Policy-Based RL

Some RL methods learn the value of actions, then act greedily. Policy gradient methods instead learn the policy directly: a function that outputs action probabilities. This handles continuous actions and stochastic policies naturally.

The Policy pi(a|s)

A parameterized policy pi(a|s, theta) maps a state to a probability distribution over actions, with parameters theta (a neural network). Training adjusts theta to favor actions that earn more reward.

class Policy(nn.Module):
    def __init__(self, obs_dim, n_actions):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 128), nn.ReLU(),
            nn.Linear(128, n_actions)
        )
    def forward(self, s):
        return torch.softmax(self.net(s), dim=-1)

Sampling Actions

Because the policy is a distribution, you sample an action rather than picking the max. Sampling provides exploration and makes the policy stochastic.

probs = policy(state)
dist = torch.distributions.Categorical(probs)
action = dist.sample()
log_prob = dist.log_prob(action)

Rolling Out a Trajectory

An episode (trajectory) is the sequence of states, actions, and rewards from start to finish. You collect a full trajectory by acting in the environment until it terminates.

states, actions, rewards, log_probs = [], [], [], []
state, _ = env.reset()
done = False
while not done:
    probs = policy(torch.tensor(state).float())
    dist = torch.distributions.Categorical(probs)
    a = dist.sample()
    state, r, term, trunc, _ = env.step(a.item())
    rewards.append(r); log_probs.append(dist.log_prob(a))
    done = term or trunc

The Discounted Return G_t

The return G_t is the total future reward from time t, discounted by gamma so nearer rewards count more. It tells us how good the actions taken from step t onward turned out.

def returns(rewards, gamma=0.99):
    G, out = 0, []
    for r in reversed(rewards):
        G = r + gamma * G
        out.insert(0, G)
    return torch.tensor(out)

The REINFORCE Objective

REINFORCE performs gradient ascent on expected return. Intuitively: increase the probability of actions that led to high return, decrease those that led to low return. Each action is weighted by its G_t.

The Policy Gradient

The loss is -sum(log_prob * G_t). The negative sign turns gradient ascent into descent so the optimizer maximizes return. Actions with high return get their log-probability pushed up.

G = returns(rewards)
loss = -torch.sum(torch.stack(log_probs) * G)

Updating the Policy

Backpropagate and step as usual. One update uses an entire trajectory, then you discard it (REINFORCE is on-policy: data from the current policy only).

optimizer.zero_grad()
loss.backward()
optimizer.step()

High Variance Problem

Vanilla REINFORCE is notoriously unstable and high-variance: returns vary wildly between episodes, so gradient estimates are noisy and learning is slow and erratic.

Baseline for Variance Reduction

Subtracting a baseline (such as the average return) from G_t reduces variance without biasing the gradient. We reward actions for being better than expected, not just for positive return.

baseline = G.mean()
advantage = G - baseline
loss = -torch.sum(torch.stack(log_probs) * advantage)

Why REINFORCE Matters

REINFORCE is the foundation of all policy gradient methods. Its weaknesses, high variance and sample inefficiency, motivate the actor-critic and PPO methods you will see next.

Quick Check

Test your policy gradient understanding.

Recap: REINFORCE

You learned to parameterize a policy pi(a|s,theta), roll out trajectories, compute the discounted return G_t, and perform gradient ascent with the loss -sum(log_prob * G_t). You saw REINFORCE's high variance and how a baseline reduces it.

Frequently asked questions

Is the “Policy Gradient Methods: REINFORCE” lesson free?

Yes — the full text of “Policy Gradient Methods: REINFORCE” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Policy Gradient Methods: REINFORCE”?

Policy gradient theorem, REINFORCE algorithm, baseline subtraction, variance reduction. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Policy Gradient Methods: REINFORCE” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Learn AI with Python