Proximal Policy Optimization (PPO)
Clipped surrogate objective, GAE advantage estimation, PPO with stable-baselines3.
Proximal Policy Optimization (PPO) is a free Learn AI with Python lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why PPO?
PPO is today's default RL algorithm: stable, sample-efficient, and easy to tune. It fixes a core problem of policy gradients, updates that are too large and destroy the policy, by limiting how much the policy can change per update.
The Danger of Big Updates
A single overly large policy update can collapse performance, and because RL is on-policy, recovery is hard. PPO keeps each update proximal (close) to the current policy to stay safe.
The Probability Ratio
PPO tracks the ratio of new to old action probabilities: ratio = pi_new(a|s) / pi_old(a|s). A ratio near 1 means the policy barely changed; far from 1 means a big shift.
ratio = torch.exp(new_log_prob - old_log_prob)The Clipped Objective
PPO's signature is the clipped surrogate objective. It multiplies the ratio by the advantage but clips the ratio to [1-eps, 1+eps], removing the incentive to move the policy too far.
clipped = torch.clamp(ratio, 1 - eps, 1 + eps)
objective = torch.min(ratio * adv, clipped * adv)
loss = -objective.mean()The Clip Threshold epsilon
eps (typically 0.2) is the clip threshold. It caps how far the ratio may move from 1, so even if the advantage is large, the policy update stays bounded and stable.
eps = 0.2 # allow at most +/-20% change in probabilityWhy min() and Clipping Work
Taking the min of the clipped and unclipped objective makes the bound pessimistic: improvements beyond the clip range give no extra reward, so the optimizer has no reason to overshoot.
Generalized Advantage Estimation
PPO estimates advantages with GAE, which blends multi-step returns using parameters gamma and lambda. GAE trades off bias and variance to give smooth, reliable advantage estimates.
def gae(rewards, values, gamma=0.99, lam=0.95):
adv, gae_acc = [], 0
for t in reversed(range(len(rewards))):
delta = rewards[t] + gamma * values[t+1] - values[t]
gae_acc = delta + gamma * lam * gae_acc
adv.insert(0, gae_acc)
return advMultiple Epochs per Batch
Unlike REINFORCE, PPO reuses each batch of collected data for several optimization epochs. The clipping makes this safe, greatly improving sample efficiency.
for _ in range(n_epochs):
# recompute ratio and clipped loss on the same batch
optimizer.zero_grad()
loss.backward()
optimizer.step()PPO with Stable-Baselines3
In practice you rarely code PPO by hand. Stable-Baselines3 provides a tested implementation. Create it with the policy type, environment, and verbosity, then call learn.
from stable_baselines3 import PPO
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100_000)Using the Trained Agent
After training, use predict to get actions for new observations. Set deterministic=True for evaluation to pick the most likely action instead of sampling.
obs, _ = env.reset()
action, _ = model.predict(obs, deterministic=True)
model.save("ppo_agent")Why PPO Dominates
PPO combines stable clipped updates, GAE advantages, and data reuse into a robust, general algorithm that works across many tasks with minimal tuning, which is why it powers everything from robotics to RLHF.
Quick Check
Test your PPO understanding.
Recap: PPO
You learned PPO's clipped surrogate objective using the probability ratio and a clip threshold eps to bound updates, GAE for advantage estimation, and data reuse over multiple epochs. You ran it easily with PPO("MlpPolicy", env, verbose=1) in Stable-Baselines3.
Frequently asked questions
Is the “Proximal Policy Optimization (PPO)” lesson free?
Yes — the full text of “Proximal Policy Optimization (PPO)” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Proximal Policy Optimization (PPO)”?
Clipped surrogate objective, GAE advantage estimation, PPO with stable-baselines3. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Proximal Policy Optimization (PPO)” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Policy Gradient Methods: REINFORCE
- Actor-Critic Methods (A2C)
- Proximal Policy Optimization (PPO)
- Custom Gymnasium Environments