6.2 Distribution shift and drift monitoring
A deployed model keeps the distribution it was trained on while the world moves on, and a drift monitor is a set of distances between a frozen reference and a sliding window that decides when someone should look.
01A model is trained on a snapshot
Training minimises the average loss over samples from one joint distribution, . Every guarantee you get from a validation score assumes that serving data come from the same distribution. Production data do not stand still: users, sensors, prices and upstream pipelines change, and the model keeps answering as if they had not.
The viewport draws that idea over time. Look first only at the front row and the white outline on it. The back row is the reference: the 1000 values of feature the model was trained on (mean 0.06, standard deviation 1.01). Every row in front of it is one monitoring window, the last samples at an evaluation every 240 samples, and 20 windows (numbered 0 to 19) stream in, newest at the front. The white ghost on each row is the reference histogram. From window 5 the mean of starts to move, and by the end of window 7 it has shifted by : the cyan histogram slides right of its white ghost.
One feature, 20 evaluations of the most recent N samples each, compared with a reference of M = 1000 training samples. A real monitor repeats this for every feature and for the predictions.
Before reading any statistic, see what no drift looks like: set the drift magnitude to and watch the stream again.
The tracks on the right score every window. Emerald bars are of (rose once they cross ), cyan bars are PSI of the predicted labels, the amber line is the KS statistic , and the grey bars are , which only appear two windows late because labels take time. Windows 7 and 8 cross the threshold (PSI 0.290 and 0.242), a retrained model goes live at window 9, its training data become the new reference, the ghosts turn emerald and PSI falls back to 0.031.
Why can the data move while accuracy holds? Think of the serving error as an average over inputs, weighted by how often each input occurs. If a region of inputs becomes twice as common in serving, its errors count twice. If the model was already accurate in the regions that grew, the average barely changes; if it was poor there, the average gets worse. Written out, that weighting is a density ratio:
Hands-On Machine Learning, Aurélien Géron, Ch. 2, Launch, monitor, and maintain your system
Professional Machine Learning Engineer exam guide, Google Cloud, Monitoring AI solutions
02Covariate, prior and concept shift
A change in the joint distribution has to show up in one of its factors, and Bayes' rule gives two ways to factor it. Which factor moved decides what a monitor can see and what the fix is.
In plain words: under covariate shift the inputs move but the rule that turns inputs into labels stays the same (more customers from a new region, same behaviour per customer). Under prior shift the share of each class moves but each class still looks the same (fraud rises from 1% to 5%, each fraud looks like before). Under concept drift the inputs look the same but the rule changes (the same browsing pattern now means a different intent). The widget shows the lab's own data before and after each kind of shift; it drives the viewport's selector and magnitude.
- 1
Both factorisations describe the same joint. A shift is named after the factor that changes while the other is held fixed.
The lab's labels follow : the true boundary is the amber parabola in the inset. The model predicts when , the best straight line under the training . The viewport is now set to concept drift with : the true boundary tilts with slope while the emerald model line stays flat.
Raise the concept drift to and keep everything else.
Switch the selector to compare. Under covariate shift the rows slide right and PSI alarms, while accuracy barely moves, because the parabola and the line part ways only at large . Under prior shift climbs from 0.51 to 0.82 and the rows hardly move; the next steps show which monitor catches it.
Maths, CS & AI Compendium, Henry Ndubuaku
AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services, Operating, monitoring, and securing ML solutions
03PSI, KS, Jensen–Shannon and chi-square
A drift statistic is a distance between two samples: the reference ( values) and the window ( values). The population stability index bins both on the reference's deciles, the nine cut points that split the reference into ten bins holding one tenth of it each, and compares the shares (reference) and (window). By construction every is 0.1; drift shows as window shares that are not. Hover to see the nine decile cuts on the newest row.
- 1
Four quartile bins, so each holds a quarter of the reference. The window has piled up in the first bin and thinned out in the last.
- 1
KL divergence on the binned shares, in both directions. Each alone is asymmetric.
The Kolmogorov–Smirnov statistic needs no bins. An empirical CDF is the share of a sample at or below : a staircase that starts at 0, climbs by at every sample value and ends at 1. KS lays the reference's staircase and the window's staircase on top of each other and measures the largest vertical gap: ( reads "the largest value over all "). Hover to draw both staircases above the newest row with the gap in amber. Its 5% critical value is here (the dashed amber line), and window 7 reaches . Because the critical value shrinks like , a large window makes tiny, harmless shifts significant, and KS applies only to continuous features.
Halve the shift and quadruple the window: , .
All of these statistics look at one feature at a time. They cannot see a change in how features move together: if and keep their histograms but their correlation flips sign, every per-feature PSI stays at its noise floor. The standard multivariate check is a domain classifier: label reference rows 0 and window rows 1, train a small classifier to tell them apart, and read its held-out AUC. An AUC near 0.5 means the two samples are indistinguishable; an AUC well above 0.5 means something moved, and the classifier's feature importances say where.
Professional Machine Learning Engineer exam guide, Google Cloud, Monitoring AI solutions: drift and skew
Maths, CS & AI Compendium, Henry Ndubuaku, Information theory; KL divergence
The rest of this lesson is in Pro
- 04Window size: detection delay versus false alarms
- 05Skew, label delay and monitoring predictions
- 06Responding: alert, diagnose, retrain, roll out
- 07Managed monitoring, and how the exams ask about it
- Python challenge: Build a drift monitor
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between data drift and concept drift?
- Data drift, or covariate shift, means the distribution of the inputs changes while the relationship between inputs and labels stays the same; it can be detected from unlabelled inputs. Concept drift means the relationship between inputs and labels changes; the inputs can look exactly as before, so it is only visible in labelled performance or label proxies. Prior shift, a change in class balance, is a third case that shows up first in the distribution of predictions.
- What PSI value indicates drift?
- The usual rule of thumb is below 0.1 for no meaningful change, 0.1 to 0.2 or 0.25 for a moderate change, and above that for a significant one. Treat these as starting points: with B bins, a window of N samples and a reference of M samples, PSI is about (B - 1)(1/N + 1/M) even when nothing changes, so small windows can cross 0.2 by chance. Check thresholds against a period you know was stable.
- What is training-serving skew?
- Training-serving skew is a difference between the features a model saw in training and the features it receives in production at the same time, caused by the system rather than the world: two implementations of a transformation, a different unit, a stale lookup or a missing join. It appears at deployment and does not go away by retraining. The fix is to compute features with shared code or a feature store for both paths and to monitor serving inputs against the training data.
Next in Production MLOps and Certification Prep
6.3Training pipelines, feature stores and servingReproducible pipelines, online/offline feature parity, and batch versus real-time serving architectures.Sources
- Professional Machine Learning Engineer exam guide, Google Cloud
- Developing AI Apps and Agents on Azure (study guide), Microsoft
- AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services
- Maths, CS & AI Compendium, Henry Ndubuaku
- Hands-On Machine Learning, Aurélien Géron