2.8 Precision, recall, ROC and calibration
A classifier only ranks cases by score; every metric in this lesson comes from where you cut that ranking, so slide one threshold through two score distributions and watch the confusion matrix, the ROC and precision-recall curves, and calibration respond.
01Scores, a threshold and four outcomes
A binary classifier does not output labels. It outputs a score per case, higher meaning more likely positive, and a separate decision rule turns the score into a label. Every metric in this lesson is a property of that rule, not of the model alone.
Start with cases you can count. Below are 20 cases, 6 positives on the top row and 14 negatives on the bottom row, each placed at its score. Drag the cyan threshold line. Every dot right of it is predicted positive, and each dot takes the colour of its cell: TP emerald, FN amber, FP rose, TN grey.
Now imagine 1000 cases instead of 20. The dots pile up so densely that each row becomes a smooth ribbon, which is what the viewport draws. A latent score is Gaussian within each class, and , and the model reports . Here and the prevalence is . The front ribbon is the positive class and the back ribbon the negative class; each ribbon's height is its density times its class count, which is why the positives are a quarter as tall.
Rows are the true class, columns the decision. The four cells sum to n, and every metric below is a ratio of cells.
The plane cuts each ribbon in two. Positives to its right are (emerald), positives to its left are (amber), negatives to its right are (rose) and negatives to its left are true negatives (grey). The area of a ribbon piece is the number of cases in it, the continuous version of counting dots. At the four areas are, per 1000 cases, TP 155, FN 45, FP 181 and TN 619. The bars on the floor are those counts, laid out like the ribbons: actual positives in front, predicted positives on the right.
Keep and double the separation to .
Hands-On Machine Learning, Aurélien Géron, Ch. 3, Confusion matrix
An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Sec. 4.4.2
02Precision, recall, F1 and specificity
The metrics differ in what they condition on. Recall and the false positive rate divide by the true class, so each looks at one ribbon. Precision divides by the decision, so it looks at everything right of the plane, both ribbons at once.
- 1
Precision is the probability that an alarm is a real positive; Bayes' rule rewrites it. P(ŷ=1 | y=1), the chance a positive is flagged, is recall written as a probability.
At the HUD reads recall , FPR (specificity ), precision , F1 and accuracy . Hover to see which regions enter it.
Move the threshold down from to .
Machine Learning Crash Course, Google, Classification: accuracy, recall, precision and related metrics
Hands-On Machine Learning, Aurélien Géron, Ch. 3, Precision and recall
03The ROC curve, and AUC as a ranking probability
Each threshold gives one point . Sweeping from 1 down to 0 traces the ROC curve from , where nothing is flagged, to , where everything is. Watch the viewport: the plane sweeps the score axis from 0 to 1 and the emerald operating point rides the curve from back to , highlighting the part it has traced, then returns to .
Before the integral, a small example. Pick one positive and one negative at random and ask: does the positive get the higher score? With three positives and three negatives there are nine pairs. Count a win as 1, a tie as and a loss as 0, and divide by 9. That fraction is the AUC.
- 1
s⁺ and s⁻ are the scores of a random positive and a random negative. With f⁻ the negative score density, d FPR = −f⁻(t) dt.
Hover : the scene draws random positive-negative pairs and links them, emerald when the positive scores higher. The running win rate settles near . Push to and the AUC becomes .
Set the separation to : positives and negatives get the same score distribution.
An Introduction to ROC Analysis, Tom Fawcett, 2006, Sec. 3, 5 and 7
Machine Learning Crash Course, Google, Classification: ROC and AUC
The rest of this lesson is in Pro
- 04Class imbalance: when accuracy and ROC mislead
- 05Choosing a threshold from costs
- 06Calibration and the Brier score
- 07More than two classes, and metrics that survive imbalance
- Python challenge: Confusion counts, ROC and AUC from scratch
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between precision and recall?
- Recall is the share of actual positives the model catches: true positives divided by all positives. Precision is the share of the model's positive predictions that are correct: true positives divided by all predicted positives. Lowering the threshold usually raises recall and lowers precision. Recall does not depend on how common positives are, while precision falls as positives become rarer.
- When should I use a precision-recall curve instead of an ROC curve?
- Use the precision-recall curve when the positive class is rare and false alarms matter, as in fraud detection, information retrieval or rare-disease screening. The ROC curve plots rates that are each computed within one class, so it looks the same at 1% and 50% prevalence and can look excellent while most alarms are false. The PR curve's baseline is the prevalence, so compare its area to that, not to 0.5.
- How do I choose a classification threshold?
- Pick it from the cost of each error on a validation set. With a false-positive cost and a false-negative cost, choose the threshold that minimises expected cost, which is where the ROC curve's slope equals (1 - prevalence) times the false-positive cost divided by prevalence times the false-negative cost. If the scores are calibrated probabilities, that is the threshold cost_FP / (cost_FP + cost_FN). Without explicit costs, fix a minimum recall or precision required by the application, or maximise an F-beta score with the beta that reflects the trade-off.
Sources
- Hands-On Machine Learning, Aurélien Géron
- An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
- Machine Learning Crash Course, Google
- An Introduction to ROC Analysis, Tom Fawcett, 2006