7.3 Policy gradients, actor-critic and PPO
Policy-gradient methods skip the value table and adjust the policy itself, making good samples more likely, and PPO's clipped ratio is what keeps those adjustments from leaping off a cliff.
01Optimising the policy directly, and why continuous actions need it
Lessons 7.1 and 7.2 learned values and read the policy off them, . That works when is one of four arrows. A robot arm chooses joint torques, a thermostat a setpoint, a trading agent a position size: the actions are real numbers, often several at once, and an argmax over a continuum is an optimisation problem of its own at every step. Policy-gradient methods skip the detour. They parameterise the policy directly and move in the direction that raises the expected return.
The viewport is the smallest problem where that matters: one decision per episode, a continuous bandit. The action is a point on the floor, and the amber surface is the reward it earns. From the start corner the ground rises toward a narrow ridge at , where peaks at 4.46. Immediately past the ridge, at , a cliff drops 3.5 to a plateau around (the rose line marks the edge). The policy is a Gaussian with a mean and a spread per dimension: the emerald ellipses mark and , and the translucent dome is its density. Its parameters are four numbers, .
Each iteration the agent tries actions drawn from (the dots, placed on the surface at the reward they earned), scores them and nudges . The rule fits in one sentence: make the good samples more likely and the bad ones less likely. Emerald dots did better than the others on average, slate dots worse, and the policy shifts toward the emerald ones. Over 80 iterations rises from 1.209 to 3.793, walks from to , and shrinks from 0.50 to the lab's floor of 0.20 as the policy reaches the ridge: there a wide policy wastes samples on the lower slopes and over the edge.
Restart and hold the first iteration, before any update.
Reinforcement Learning: An Introduction, Sutton & Barto, Ch. 13 intro and 13.1, Policy approximation and its advantages
Hands-On Machine Learning, Aurélien Géron, Ch. 18, Policy gradients
02The policy-gradient theorem and REINFORCE
To climb you need , but is an integral over actions and is a black box: the environment hands back a number, never a derivative. The log-derivative trick moves the gradient off and onto the policy, which you wrote yourself and can differentiate. Check it with numbers first. In the widget the action is one-dimensional, the reward is a bump at , and the policy is .
- 1
R(a) does not depend on θ, so the gradient only touches the policy.
For episodes with many steps, apply the same argument to a whole trajectory . Its log-probability is . The dynamics do not depend on , so their gradient vanishes and you never need a model of the environment. Rewards that arrived before cannot depend on it, so each action is credited only with the return that follows it:
For the Gaussian policy the score has a closed form. With :
Weighted by the reward, the first formula pulls toward samples that scored well, and the second widens the policy when the good samples lie far out and narrows it when they lie close in. On the ridge the high rewards sit in a band 0.2 wide around , so the narrowing wins along : that is falling to 0.20 in the viewport while , along the ridge, stays at 0.29.
Actions a (N × 2), rewards (N), score ∇θ log π(aᵢ) (N × 4: two for μ, two for log σ). The estimate ĝ is the reward-weighted mean over the N rows, a vector of 4.
Reinforcement Learning: An Introduction, Sutton & Barto, Ch. 13.2 and 13.3, The policy gradient theorem; REINFORCE
Hands-On Machine Learning, Aurélien Géron, Ch. 18, Policy gradients
03Variance, baselines and the advantage
is unbiased but noisy, and the viewport measures how noisy. At every iteration it draws 32 more batches at the same and plots their estimates as small amber dots around the arrow's tip. The amber ring's radius is their spread, , the typical distance between one batch's estimate and the average; the grey ring is the spread with the baseline toggled the other way. At the start, with , the spread is 0.095 with the baseline and 0.572 without, while the true gradient has length 0.27. Without a baseline, one batch's estimate is mostly noise.
The reason is that every reward here is positive: 1.06 to 1.59 in the first batch. With every sample says "more of this", each pulls toward itself, and is the small difference between large pulls in all directions. Subtracting a baseline from every reward keeps only what differs between samples.
- 1
The log-derivative trick, run backwards.
What remains is the advantage : how much better than usual a sample did. In multi-step problems the baseline may depend on the state, , and the weight becomes the advantage function from lessons 7.1 and 7.2: positive for actions better than the policy's average in that state.
Turn the baseline off and hold the first batch.
Keep the baseline off and let it train with the same and .
No baseline, but : hold the first batch.
Reinforcement Learning: An Introduction, Sutton & Barto, Ch. 13.4, REINFORCE with baseline
The rest of this lesson is in Pro
- 04Actor-critic and GAE, in brief
- 05Trust regions and PPO's clipped surrogate
- 06PPO in practice, its role in RLHF, and where RL is used
- Python challenge: Score functions, REINFORCE with a baseline and PPO's clip
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between REINFORCE and PPO?
- Both follow the policy gradient: make actions with a positive advantage more likely and actions with a negative advantage less likely. REINFORCE takes one gradient step per batch of samples and then throws the batch away. PPO reuses each batch for several epochs of minibatch steps on a surrogate objective whose probability ratio is clipped, so a sample stops contributing once the new policy has moved its probability by more than a set factor. That bounds how far each update can move the policy, which makes PPO more sample-efficient and much more stable. PPO also usually learns a critic for its advantages, while plain REINFORCE uses returns with at most a simple baseline.
- Why does PPO clip the probability ratio?
- The surrogate objective uses advantages measured under the old policy, and it is only accurate while the new policy stays close to the old one. Clipping the ratio between 1 minus epsilon and 1 plus epsilon removes the incentive to push any single action's probability further once it has moved by that factor, a cheap first-order version of TRPO's trust region. Taking the minimum with the unclipped term means the clip never hides an update that made a bad action more likely or a good action less likely. Epsilon is usually 0.1 to 0.3, with 0.2 the common default.
- What is an advantage function in reinforcement learning?
- The advantage A(s, a) = Q(s, a) − V(s) measures how much better taking action a in state s is than what the policy does on average in that state. Policy-gradient methods weight each action's log-probability gradient by its advantage instead of by the raw return, which leaves the expected gradient unchanged but removes the part of the return that every action would have received anyway, cutting the variance sharply. In practice it is estimated from a learned value function, for example with generalised advantage estimation (GAE).
Sources
- Reinforcement Learning: An Introduction, Sutton & Barto
- Proximal Policy Optimization Algorithms, Schulman et al., 2017
- Hands-On Machine Learning, Aurélien Géron
- Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022