0Pricing
Learn AI with Python · Lesson

Actor-Critic Methods (A2C)

Advantage function, actor (policy) + critic (value), synchronous A2C implementation.

Actor-Critic Methods (A2C) is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Combining Policy and Value

Actor-Critic methods merge two ideas: an actor (a policy that chooses actions) and a critic (a value function that judges them). The critic gives lower-variance feedback than raw returns, stabilizing learning.

The Actor

The actor is the policy network. It outputs action logits (or probabilities) for the current state, exactly like in REINFORCE, and decides what to do.

class ActorCritic(nn.Module):
    def __init__(self, obs_dim, n_actions):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU())
        self.actor = nn.Linear(128, n_actions)

The Critic

The critic estimates the state value V(s): the expected return from state s. It is a single-output head that predicts how good the current situation is.

        self.critic = nn.Linear(128, 1)

    def forward(self, s):
        h = self.shared(s)
        return self.actor(h), self.critic(h)

Shared Backbone, Two Heads

Actor and critic usually share the early layers (the backbone) and split into two heads. Sharing features is efficient and lets both tasks reinforce useful representations.

The Advantage Function

The key quantity is the advantage A = r + gamma * V(s') - V(s). It measures how much better an action was than the critic expected. Positive advantage means "better than average", negative means worse.

advantage = reward + gamma * V_next - V_current

Why Advantage Beats Raw Return

Using advantage instead of the full return G_t centers the signal around the critic's estimate, dramatically cutting variance. This is the main reason actor-critic learns more stably than REINFORCE.

The Actor Loss

The actor is trained like REINFORCE but weighted by the advantage instead of the return: increase the probability of actions with positive advantage.

actor_loss = -(log_prob * advantage.detach()).mean()

The Critic Loss

The critic learns to predict returns accurately. Its loss is the squared error between its prediction V(s) and the observed target r + gamma * V(s').

target = reward + gamma * V_next.detach()
critic_loss = (V_current - target).pow(2).mean()

Combined Loss

Train both heads together by summing the losses (often with a small entropy bonus to encourage exploration). One backward pass updates the shared backbone and both heads.

loss = actor_loss + 0.5 * critic_loss - 0.01 * entropy
loss.backward()

Synchronous A2C

A2C (Advantage Actor-Critic) is the synchronous version: multiple parallel workers collect experience from copies of the environment simultaneously, then a single synchronized update uses all their data, improving stability and throughput.

# N parallel envs step together each iteration
# gather all transitions, then one combined update

A2C vs A3C

The asynchronous variant A3C lets workers update independently. A2C synchronizes them, which is simpler, more GPU-friendly, and usually just as effective, hence its popularity.

Quick Check

Test your actor-critic understanding.

Recap: Actor-Critic and A2C

You learned the actor outputs policy logits and the critic estimates V(s), often via a shared backbone with two heads. The advantage r + gamma*V(s') - V(s) reduces variance, and synchronous A2C uses parallel workers for stable, efficient training.

Frequently asked questions

Is the “Actor-Critic Methods (A2C)” lesson free?

Yes — the full text of “Actor-Critic Methods (A2C)” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Actor-Critic Methods (A2C)”?

Advantage function, actor (policy) + critic (value), synchronous A2C implementation. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Actor-Critic Methods (A2C)” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Policy Gradient Methods: REINFORCE
  2. Actor-Critic Methods (A2C)
  3. Proximal Policy Optimization (PPO)
  4. Custom Gymnasium Environments
← Back to Learn AI with Python