Skip to content
LearnMLByDoing
Free

7.3 Policy gradients, actor-critic and PPO

36 minProadvancedS&BPPOHOMLInstructGPT

Policy-gradient methods skip the value table and adjust the policy itself, making good samples more likely, and PPO's clipped ratio is what keeps those adjustments from leaping off a cliff.

01Optimising the policy directly, and why continuous actions need it

Lessons 7.1 and 7.2 learned values and read the policy off them, π(s)=arg⁡max⁡aQ(s,a)\pi(s) = \arg\max_a Q(s,a). That works when aa is one of four arrows. A robot arm chooses joint torques, a thermostat a setpoint, a trading agent a position size: the actions are real numbers, often several at once, and an argmax over a continuum is an optimisation problem of its own at every step. Policy-gradient methods skip the detour. They parameterise the policy πθ(a∣s)\tok{policy}{\pi_\theta}(a\mid s) directly and move θ\theta in the direction that raises the expected return.

The viewport is the smallest problem where that matters: one decision per episode, a continuous bandit. The action a=(a1,a2)a = (a_1, a_2) is a point on the floor, and the amber surface is the reward R(a)\tok{reward}{R}(a) it earns. From the start corner the ground rises toward a narrow ridge at a1=1.3a_1 = 1.3, where RR peaks at 4.46. Immediately past the ridge, at a1=1.62a_1 = 1.62, a cliff drops 3.5 to a plateau around −1-1 (the rose line marks the edge). The policy is a Gaussian with a mean μ\tok{mu}{\mu} and a spread σ\tok{sigma}{\sigma} per dimension: the emerald ellipses mark 1σ1\sigma and 2σ2\sigma, and the translucent dome is its density. Its parameters are four numbers, θ=(μ1,μ2,log⁡σ1,log⁡σ2)\theta = (\mu_1, \mu_2, \log\sigma_1, \log\sigma_2).

πθ(a)=N(a; μ, diag(σ2)),J(θ)=Ea∼πθ[R(a)]=∫πθ(a) R(a) da\tok{policy}{\pi_\theta}(a)=\mathcal{N}\big(a;\ \tok{mu}{\mu},\ \mathrm{diag}(\tok{sigma}{\sigma}^2)\big),\qquad \tok{objective}{J}(\theta)=\mathbb{E}_{a\sim\pi_\theta}\big[\tok{reward}{R}(a)\big]=\int \pi_\theta(a)\,R(a)\,da
The objective is the reward averaged over the policy's own actions. Gradient ascent on J is the whole method.

Each iteration the agent tries N=32N = 32 actions drawn from πθ\pi_\theta (the dots, placed on the surface at the reward they earned), scores them and nudges θ\theta. The rule fits in one sentence: make the good samples more likely and the bad ones less likely. Emerald dots did better than the others on average, slate dots worse, and the policy shifts toward the emerald ones. Over 80 iterations JJ rises from 1.209 to 3.793, μ\mu walks from (−1.60,−1.40)(-1.60, -1.40) to (1.16,0.55)(1.16, 0.55), and σ1\sigma_1 shrinks from 0.50 to the lab's floor of 0.20 as the policy reaches the ridge: there a wide policy wastes samples on the lower slopes and over the edge.

Restart and hold the first iteration, before any update.

Reinforcement Learning: An Introduction, Sutton & Barto, Ch. 13 intro and 13.1, Policy approximation and its advantages

Hands-On Machine Learning, Aurélien Géron, Ch. 18, Policy gradients

02The policy-gradient theorem and REINFORCE

To climb JJ you need ∇θJ\nabla_\theta J, but JJ is an integral over actions and RR is a black box: the environment hands back a number, never a derivative. The log-derivative trick moves the gradient off RR and onto the policy, which you wrote yourself and can differentiate. Check it with numbers first. In the widget the action is one-dimensional, the reward is a bump at a=1a = 1, and the policy is N(μ,0.82)\mathcal{N}(\mu, 0.8^2).

Loading figure
Top readouts: at the probe a=0a = 0 with μ=−1\mu = -1, nudging μ\mu changes π(a)\pi(a) at the rate 0.3567, and π(a)⋅(a−μ)/σ2=0.2283×1.5625\pi(a)\cdot(a-\mu)/\sigma^2 = 0.2283 \times 1.5625 gives the same 0.3567. Drag the probe anywhere: the two always agree. Bottom readouts: the exact slope is dJ/dμ=+0.563dJ/d\mu = +0.563. Batch #1 estimates it as +1.091+1.091, and 400 batches average +0.568+0.568 with a spread of 0.589 per batch. Each sample's term is its reward times (a−μ)/σ2(a-\mu)/\sigma^2: samples right of μ\mu pull it right, and the pull grows with the reward. Turn the baseline on before step 3 explains it.
The log-derivative trick1/4
  1. 1
    ∇θJ=∇θ∫πθ(a) R(a) da=∫∇θπθ(a) R(a) da\nabla_\theta \tok{objective}{J}=\nabla_\theta\int \pi_\theta(a)\,R(a)\,da=\int \nabla_\theta\pi_\theta(a)\,R(a)\,da

    R(a) does not depend on θ, so the gradient only touches the policy.

For episodes with many steps, apply the same argument to a whole trajectory τ=(s0,a0,r1,s1,… )\tau = (s_0, a_0, r_1, s_1, \dots). Its log-probability is log⁡p(s0)+∑t[log⁡πθ(at∣st)+log⁡P(st+1∣st,at)]\log p(s_0) + \sum_t [\log\pi_\theta(a_t\mid s_t) + \log P(s_{t+1}\mid s_t, a_t)]. The dynamics PP do not depend on θ\theta, so their gradient vanishes and you never need a model of the environment. Rewards that arrived before ata_t cannot depend on it, so each action is credited only with the return that follows it:

∇θJ(θ)=Eτ∼πθ[∑t∇θlog⁡πθ(at∣st) Gt],Gt=∑k≥0γkrt+k+1\nabla_\theta \tok{objective}{J}(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\Big[\sum_{t}\tok{score}{\nabla_\theta\log\pi_\theta}(a_t\mid s_t)\,G_t\Big],\qquad G_t=\sum_{k\ge 0}\gamma^k r_{t+k+1}
The policy-gradient theorem in its REINFORCE form. In the viewport each episode is one step, so G is the reward R(a).

For the Gaussian policy the score has a closed form. With log⁡πθ(a)=−∑d[(ad−μd)2/(2σd2)+log⁡σd]+const\log\pi_\theta(a) = -\sum_d \big[(a_d-\mu_d)^2/(2\sigma_d^2) + \log\sigma_d\big] + \text{const}:

∂log⁡πθ(a)∂μd=ad−μdσd2,∂log⁡πθ(a)∂log⁡σd=(ad−μd)2σd2−1\frac{\partial\log\pi_\theta(a)}{\partial\mu_d}=\frac{a_d-\tok{mu}{\mu}_d}{\tok{sigma}{\sigma}_d^{2}},\qquad \frac{\partial\log\pi_\theta(a)}{\partial\log\sigma_d}=\frac{(a_d-\mu_d)^2}{\sigma_d^{2}}-1
The μ-gradient points from μ toward the sample. The log σ-gradient is positive for a sample more than 1σ out and negative inside.

Weighted by the reward, the first formula pulls μ\mu toward samples that scored well, and the second widens the policy when the good samples lie far out and narrows it when they lie close in. On the ridge the high rewards sit in a band 0.2 wide around a1=1.3a_1 = 1.3, so the narrowing wins along a1a_1: that is σ1\sigma_1 falling to 0.20 in the viewport while σ2\sigma_2, along the ridge, stays at 0.29.

One REINFORCE batch[,]128 values

Actions a (N × 2), rewards (N), score ∇θ log π(aᵢ) (N × 4: two for μ, two for log σ). The estimate ĝ is the reward-weighted mean over the N rows, a vector of 4.

Reinforcement Learning: An Introduction, Sutton & Barto, Ch. 13.2 and 13.3, The policy gradient theorem; REINFORCE

Hands-On Machine Learning, Aurélien Géron, Ch. 18, Policy gradients

03Variance, baselines and the advantage

g^\hat g is unbiased but noisy, and the viewport measures how noisy. At every iteration it draws 32 more batches at the same θ\theta and plots their estimates as small amber dots around the arrow's tip. The amber ring's radius is their spread, tr⁡Cov⁡[g^μ]\sqrt{\operatorname{tr}\operatorname{Cov}[\hat g_\mu]}, the typical distance between one batch's estimate and the average; the grey ring is the spread with the baseline toggled the other way. At the start, with N=32N = 32, the spread is 0.095 with the baseline and 0.572 without, while the true gradient (0.218,0.167)(0.218, 0.167) has length 0.27. Without a baseline, one batch's estimate is mostly noise.

The reason is that every reward here is positive: 1.06 to 1.59 in the first batch. With b=0b = 0 every sample says "more of this", each pulls μ\mu toward itself, and g^\hat g is the small difference between large pulls in all directions. Subtracting a baseline b\tok{baseline}{b} from every reward keeps only what differs between samples.

A baseline adds no bias1/4
  1. 1
    Ea∼πθ[∇θlog⁡πθ(a)]=∫πθ(a)∇θπθ(a)πθ(a) da=∫∇θπθ(a) da\mathbb{E}_{a\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a)\big]=\int\pi_\theta(a)\frac{\nabla_\theta\pi_\theta(a)}{\pi_\theta(a)}\,da=\int\nabla_\theta\pi_\theta(a)\,da

    The log-derivative trick, run backwards.

What remains is the advantage A=R−b\tok{advantage}{A} = R - b: how much better than usual a sample did. In multi-step problems the baseline may depend on the state, b=V(st)b = V(s_t), and the weight becomes the advantage function A(s,a)=Q(s,a)−V(s)A(s,a) = Q(s,a) - V(s) from lessons 7.1 and 7.2: positive for actions better than the policy's average in that state.

∇θJ=E[∑t∇θlog⁡πθ(at∣st) A(st,at)],A(s,a)=Q(s,a)−V(s)\nabla_\theta \tok{objective}{J}=\mathbb{E}\Big[\sum_t \tok{score}{\nabla_\theta\log\pi_\theta}(a_t\mid s_t)\,\tok{advantage}{A}(s_t,a_t)\Big],\qquad A(s,a)=Q(s,a)-V(s)

Turn the baseline off and hold the first batch.

Keep the baseline off and let it train with the same η=0.1\eta = 0.1 and N=32N = 32.

No baseline, but N=128N = 128: hold the first batch.

Reinforcement Learning: An Introduction, Sutton & Barto, Ch. 13.4, REINFORCE with baseline

The rest of this lesson is in Pro

  • 04Actor-critic and GAE, in brief
  • 05Trust regions and PPO's clipped surrogate
  • 06PPO in practice, its role in RLHF, and where RL is used
  • Python challenge: Score functions, REINFORCE with a baseline and PPO's clip
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between REINFORCE and PPO?
Both follow the policy gradient: make actions with a positive advantage more likely and actions with a negative advantage less likely. REINFORCE takes one gradient step per batch of samples and then throws the batch away. PPO reuses each batch for several epochs of minibatch steps on a surrogate objective whose probability ratio is clipped, so a sample stops contributing once the new policy has moved its probability by more than a set factor. That bounds how far each update can move the policy, which makes PPO more sample-efficient and much more stable. PPO also usually learns a critic for its advantages, while plain REINFORCE uses returns with at most a simple baseline.
Why does PPO clip the probability ratio?
The surrogate objective uses advantages measured under the old policy, and it is only accurate while the new policy stays close to the old one. Clipping the ratio between 1 minus epsilon and 1 plus epsilon removes the incentive to push any single action's probability further once it has moved by that factor, a cheap first-order version of TRPO's trust region. Taking the minimum with the unclipped term means the clip never hides an update that made a bad action more likely or a good action less likely. Epsilon is usually 0.1 to 0.3, with 0.2 the common default.
What is an advantage function in reinforcement learning?
The advantage A(s, a) = Q(s, a) − V(s) measures how much better taking action a in state s is than what the policy does on average in that state. Policy-gradient methods weight each action's log-probability gradient by its advantage instead of by the raw return, which leaves the expected gradient unchanged but removes the part of the return that every action would have received anyway, cutting the variance sharply. In practice it is estimated from a learned value function, for example with generalised advantage estimation (GAE).

Sources

  • Reinforcement Learning: An Introduction, Sutton & Barto
  • Proximal Policy Optimization Algorithms, Schulman et al., 2017
  • Hands-On Machine Learning, Aurélien Géron
  • Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022