4.6 Alignment: reward models, RLHF and DPO
Preference tuning pushes a fine-tuned model toward answers people prefer while a KL leash keeps it near where it started, and the leash's strength decides whether it learns to help or learns to game the reward model.
01Pretraining, SFT, and what a policy is
A chat model is built in three stages. Pretraining teaches a base model to continue web text by next-token prediction (lesson 4.4). Supervised fine-tuning (SFT) trains it further on tens of thousands of written demonstrations, prompts paired with good answers, so it answers instead of continuing. Preference tuning, the subject of this lesson, then teaches it which of its own answers people prefer.
In reinforcement-learning language the model is a policy: given a prompt it puts a probability on every complete response . That probability is the product of its next-token probabilities, so its log is a sum you can compute in one forward pass:
Gather the log-probability of each actual next token from the logits, mask the prompt and padding, then sum over T to get one number per response, shape [B]. RLHF and DPO both work with these sums.
The viewport shrinks this to one prompt and 625 candidate responses, laid out on a floor by two features: , how much substance a response has, and , how long and flattering it is. Heights are , so a policy that piles all its mass on one response still fits on screen. The emerald surface starts as the broad pretrained model and SFT pulls it onto the cyan wire, , while cyan demonstrations land on the floor. The HUD's falls from 0.41 nats to 0.00 by step 120.
SFT can only copy what demonstrators wrote. It cannot express "this answer is better than that one", and writing ideal answers is slower than comparing two. The SFT model is where preference tuning starts, and it stays in the picture as , the anchor that every later step is measured against.
Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022, Sec. 3.1 and 3.5
Maths, CS & AI Compendium, Henry Ndubuaku, LLM post-training
02Preference data: which answer is better?
A preference dataset is built by sampling two or more responses to the same prompt, usually from the SFT model, and asking a rater which is better. One record looks like this:
prompt Can I take ibuprofen with my blood-pressure medication? chosen Ibuprofen can raise blood pressure and weaken some blood-pressure medicines, so ask your pharmacist or doctor before combining them. rejected Yes, ibuprofen is safe to take with any medication.
InstructGPT's reward model was trained on roughly 33k prompts, each with 4 to 9 ranked responses; a ranking of K responses yields K(K−1)/2 pairs.
Comparisons are used instead of scores out of ten because people disagree about absolute scales but agree more often about which of two answers is better. They still disagree a lot: InstructGPT reports that its labelers agreed with each other about 73% of the time. Labels are noisy, and the model of them has to allow for that.
The viewport is running RLHF, which starts with this data. Its 400 pairs are drawn from , so they cluster under the cyan wire; each is a floor segment with an emerald chosen end and a grey rejected end, and 24 of them are on screen at a time. Their labels come from the raters' preferences in the toy: the share of raters choosing over is , so pairs near the reference are close calls, between 50% and 73% agreement.
Restart the RLHF run and hold it at step 0 to read the data before any training.
Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022, Sec. 3.3 and 3.4
03Reward models and the Bradley–Terry loss
A reward model is usually the SFT network with its vocabulary head replaced by a single scalar output. It is trained so that the difference of two scores predicts the preference, through the Bradley–Terry model:
- 1
Maximum likelihood of the observed choices: the negative log-probability the model gave to what the rater chose.
In the viewport, RLHF stage 1 of 2 fits the reward model for 60 steps while the policy waits. The toy reward model is a weighted sum of two features of a response: a broad helpful hill at and a narrow bump at , long and flattering answers. The fit recovers weights for the raters' , and the amber surface grows from flat into two hills; the bump is the artefact peak. The loss only falls from 0.693 to 0.681, because pairs drawn near are close calls and their labels are close to 50/50.
Restart RLHF with and watch the first two seconds.
Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022, Sec. 3.5, Reward modeling
Maths, CS & AI Compendium, Henry Ndubuaku, Logistic regression and the sigmoid
The rest of this lesson is in Pro
- 04RLHF: maximise reward on a KL leash
- 05The closed-form optimum, and DPO
- 06Reward hacking, over-optimisation and the role of β
- 07Variants and practice
- Python challenge: Bradley–Terry, DPO and the KL-optimal policy
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between RLHF and DPO?
- Both optimise the same objective: expected reward minus a KL penalty that keeps the model close to its supervised starting point. RLHF first trains a separate reward model on human preference pairs and then uses reinforcement learning, usually PPO, to sample responses and push the model toward high scores. DPO uses the closed-form solution of that objective to rewrite the reward in terms of the model's own probabilities, so it trains directly on the preference pairs with a supervised-style loss, with no reward model, no sampling and no value network. RLHF learns from the model's own new outputs; DPO is simpler and cheaper but learns only from the fixed dataset.
- What does the KL penalty (beta) do in RLHF?
- It charges the model for drifting away from the reference model it started from. The reward model is only reliable on the kind of answers it was trained on, and the reference model writes those answers, so the penalty keeps the policy where the reward means something and preserves its fluency and knowledge. A large beta keeps the model almost unchanged; a small beta lets it chase the reward model's highest scores, including its mistakes. The optimal policy for a given beta is the reference model reweighted by exp(reward / beta).
- What is reward hacking in language models?
- Reward hacking, or reward over-optimisation, happens when a model finds responses that a learned reward model scores highly but people do not actually prefer, such as overly long, flattering or evasive answers. The reward-model score keeps rising while real quality falls, which is Goodhart's law. It is controlled with a stronger KL penalty or early stopping, better and fresher preference data, reward-model ensembles, length control, and evaluation by humans or independent judges on held-out prompts.
Next in Transformers and LLMs
4.7LLM agents, tool use and evaluationReason, act, observe: the agent loop, function calling, planning budgets, and how to evaluate a system that takes actions.Sources
- Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022
- Direct Preference Optimization, Rafailov et al., 2023
- Maths, CS & AI Compendium, Henry Ndubuaku
- Proximal Policy Optimization Algorithms, Schulman et al., 2017