07
Reinforcement Learning
Agents that learn from reward: Bellman equations, value iteration, Q-learning and the policy-gradient methods behind RLHF.
- Lessons
- 3
- Study time
- 1.7 h
- Access
- Pro track
Syllabus
- 7.1
MDPs, Bellman equations and value iteration
States, actions, rewards and discounting; watch value propagate backwards across a gridworld until the greedy policy stops changing.
34 minProLive 3D lab and Python challengeintermediate- Formalise a problem as a Markov decision process
- Derive the Bellman optimality equation and run value iteration
- Extract a greedy policy and explain the role of the discount factor
- 7.2
Q-learning, SARSA and exploration
Learn action values from experience alone, balance exploration against exploitation, and see why off-policy learning walks the cliff edge.
34 minProLive 3D lab and Python challengeintermediate- Implement tabular Q-learning with an epsilon-greedy policy
- Contrast on-policy SARSA with off-policy Q-learning
- Explain how DQN stabilises learning with replay and target networks
- 7.3
Policy gradients, actor-critic and PPO
Differentiate expected reward directly: REINFORCE, baselines and advantages, and the clipped objective that made PPO the default.
36 minProLive 3D lab and Python challengeadvanced- Derive the policy-gradient theorem and the REINFORCE estimator
- Reduce variance with baselines and advantage estimates
- Explain PPO's clipped surrogate objective and its link to RLHF