Skip to content
LearnMLByDoing
Free

6.2 Distribution shift and drift monitoring

28 minProintermediateGCP-PMLEAI-103MLA-C02CompendiumHOML

A deployed model keeps the distribution it was trained on while the world moves on, and a drift monitor is a set of distances between a frozen reference and a sliding window that decides when someone should look.

01A model is trained on a snapshot

Training minimises the average loss over samples from one joint distribution, Ptrain(x,y)P_{\text{train}}(x, y). Every guarantee you get from a validation score assumes that serving data come from the same distribution. Production data do not stand still: users, sensors, prices and upstream pipelines change, and the model keeps answering as if they had not.

The viewport draws that idea over time. Look first only at the front row and the white outline on it. The back row is the reference: the 1000 values of feature x1x_1 the model was trained on (mean 0.06, standard deviation 1.01). Every row in front of it is one monitoring window, the last N=200\tok{window}{N}=200 samples at an evaluation every 240 samples, and 20 windows (numbered 0 to 19) stream in, newest at the front. The white ghost on each row is the reference histogram. From window 5 the mean of P(x)\tok{p_x}{P(x)} starts to move, and by the end of window 7 it has shifted by +0.6+0.6: the cyan histogram slides right of its white ghost.

monitoring stream[,]4,000 values

One feature, 20 evaluations of the most recent N samples each, compared with a reference of M = 1000 training samples. A real monitor repeats this for every feature and for the predictions.

Before reading any statistic, see what no drift looks like: set the drift magnitude to m=0m=0 and watch the stream again.

The tracks on the right score every window. Emerald bars are PSI\tok{psi}{\mathrm{PSI}} of x1x_1 (rose once they cross τ=0.2\tok{tau}{\tau}=0.2), cyan bars are PSI of the predicted labels, the amber line is the KS statistic D\tok{ks}{D}, and the grey bars are acc\tok{accuracy}{\mathrm{acc}}, which only appear two windows late because labels take time. Windows 7 and 8 cross the threshold (PSI 0.290 and 0.242), a retrained model goes live at window 9, its training data become the new reference, the ghosts turn emerald and PSI falls back to 0.031.

Why can the data move while accuracy holds? Think of the serving error as an average over inputs, weighted by how often each input occurs. If a region of inputs becomes twice as common in serving, its errors count twice. If the model was already accurate in the regions that grew, the average barely changes; if it was poor there, the average gets worse. Written out, that weighting is a density ratio:

Rserve(f)=EPserve[ℓ(f(x),y)]=EPtrain ⁣[Pserve(x,y)Ptrain(x,y) ℓ(f(x),y)]R_{\text{serve}}(f)=\mathbb{E}_{P_{\text{serve}}}\big[\ell(f(x),y)\big]=\mathbb{E}_{P_{\text{train}}}\!\left[\frac{P_{\text{serve}}(x,y)}{P_{\text{train}}(x,y)}\,\ell(f(x),y)\right]
Serving risk is training risk reweighted by the density ratio. Where the ratio is far from 1, the validation score says nothing.

Hands-On Machine Learning, Aurélien Géron, Ch. 2, Launch, monitor, and maintain your system

Professional Machine Learning Engineer exam guide, Google Cloud, Monitoring AI solutions

02Covariate, prior and concept shift

A change in the joint distribution has to show up in one of its factors, and Bayes' rule gives two ways to factor it. Which factor moved decides what a monitor can see and what the fix is.

In plain words: under covariate shift the inputs move but the rule that turns inputs into labels stays the same (more customers from a new region, same behaviour per customer). Under prior shift the share of each class moves but each class still looks the same (fraud rises from 1% to 5%, each fraud looks like before). Under concept drift the inputs look the same but the rule changes (the same browsing pattern now means a different intent). The widget shows the lab's own data before and after each kind of shift; it drives the viewport's selector and magnitude.

Loading figure
Cyan points are y=1y=1, grey y=0y=0; amber is the true boundary, emerald the deployed model x2>0x_2>0, white ticks mark the rates before the drift. At m=0.6m=0.6, covariate: the mean of x1x_1 moves from 0.00 to 0.60 and the fixed model's accuracy only from 0.868 to 0.858. Prior: P(y=1)P(y{=}1) jumps from 0.512 to 0.832 and the predicted rate from 0.497 to 0.745, while the x1x_1 marginal stays on its ghost. Concept: nothing in the cloud moves, but accuracy falls from 0.868 to 0.787. Push mm to 1.5 under covariate shift and the accuracy reaches 0.777: far enough out, the parabola and the line do part.
What changes in each kind of shift1/4
  1. 1
    P(x,y)=P(y∣x) P(x)=P(x∣y) P(y)P(x,y)=\tok{p_y_x}{P(y\mid x)}\,\tok{p_x}{P(x)}=P(x\mid y)\,\tok{p_y}{P(y)}

    Both factorisations describe the same joint. A shift is named after the factor that changes while the other is held fixed.

The lab's labels follow P(y=1∣x)=σ(8 (x2−0.3x12+0.3))P(y{=}1\mid x)=\sigma\big(8\,(x_2-0.3x_1^2+0.3)\big): the true boundary is the amber parabola in the inset. The model predicts y=1y=1 when x2>0x_2>0, the best straight line under the training P(x1)P(x_1). The viewport is now set to concept drift with m=0.6m=0.6: the true boundary tilts with slope 1.2m=0.721.2m=0.72 while the emerald model line stays flat.

Raise the concept drift to m=1.5m=1.5 and keep everything else.

Switch the selector to compare. Under covariate shift the rows slide right and PSI alarms, while accuracy barely moves, because the parabola and the line part ways only at large ∣x1∣|x_1|. Under prior shift P(y=1)P(y{=}1) climbs from 0.51 to 0.82 and the rows hardly move; the next steps show which monitor catches it.

Maths, CS & AI Compendium, Henry Ndubuaku

AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services, Operating, monitoring, and securing ML solutions

03PSI, KS, Jensen–Shannon and chi-square

A drift statistic is a distance between two samples: the reference (M=1000M=1000 values) and the window (NN values). The population stability index bins both on the reference's deciles, the nine cut points that split the reference into ten bins holding one tenth of it each, and compares the shares eie_i (reference) and aia_i (window). By construction every eie_i is 0.1; drift shows as window shares that are not. Hover PSI\tok{psi}{\mathrm{PSI}} to see the nine decile cuts on the newest row.

Worked example: four bins1/4
  1. 1
    e=(0.25, 0.25, 0.25, 0.25),a=(0.4, 0.3, 0.2, 0.1)e=(0.25,\ 0.25,\ 0.25,\ 0.25),\qquad a=(0.4,\ 0.3,\ 0.2,\ 0.1)

    Four quartile bins, so each holds a quarter of the reference. The window has piled up in the first bin and thinned out in the last.

PSI=∑i=1B(ai−ei)ln⁡aiei\tok{psi}{\mathrm{PSI}}=\sum_{i=1}^{B}(a_i-e_i)\ln\frac{a_i}{e_i}
The same sum for B = 10 bins. Empty bins are floored at ε = 10⁻⁴ so the logarithm stays finite.
PSI is a symmetrised KL divergence1/5
  1. 1
    KL(a ∥ e)=∑iailn⁡aiei,KL(e ∥ a)=∑ieiln⁡eiai=−∑ieiln⁡aiei\mathrm{KL}(a\,\|\,e)=\sum_i a_i\ln\frac{a_i}{e_i},\qquad \mathrm{KL}(e\,\|\,a)=\sum_i e_i\ln\frac{e_i}{a_i}=-\sum_i e_i\ln\frac{a_i}{e_i}

    KL divergence on the binned shares, in both directions. Each alone is asymmetric.

The Kolmogorov–Smirnov statistic needs no bins. An empirical CDF F(x)F(x) is the share of a sample at or below xx: a staircase that starts at 0, climbs by 1/N1/N at every sample value and ends at 1. KS lays the reference's staircase and the window's staircase on top of each other and measures the largest vertical gap: D=sup⁡x∣Fref(x)−Fwin(x)∣\tok{ks}{D}=\sup_x|F_{\text{ref}}(x)-F_{\text{win}}(x)| (sup⁡\sup reads "the largest value over all xx"). Hover D\tok{ks}{D} to draw both staircases above the newest row with the gap in amber. Its 5% critical value is 1.36(M+N)/(MN)=0.1051.36\sqrt{(M+N)/(MN)}=0.105 here (the dashed amber line), and window 7 reaches D=0.194D=0.194. Because the critical value shrinks like 1/N1/\sqrt N, a large window makes tiny, harmless shifts significant, and KS applies only to continuous features.

Loading figure
The reference is the lab's 1000-sample training snapshot; the window is NN fresh draws of N(δ,σ2)\mathcal N(\delta,\sigma^2). At δ=0.6\delta=0.6, N=200N=200 the window piles into the upper bins: PSI 0.259 and D=0.236D=0.236 against a critical value of 0.105. Set δ=0\delta=0: PSI falls to 0.059, right at the no-drift floor of 0.054, and DD to 0.095. Now drag NN down to 50: with no drift at all PSI is 0.130 and the floor 0.189; raise the bins to 20 and PSI jumps to 0.671: one bin is left empty, and that bin alone adds 0.310 (the tallest emerald term). Numbers are for the first draw; resample shows how much they move by chance.

Halve the shift and quadruple the window: m=0.3m=0.3, N=800N=800.

JS(a,e)=12KL(a ∥ m)+12KL(e ∥ m),  m=a+e2;χ2=∑i(Oi−Ei)2Ei,  Ei=Nei\mathrm{JS}(a,e)=\tfrac12\mathrm{KL}(a\,\|\,m)+\tfrac12\mathrm{KL}(e\,\|\,m),\ \ m=\tfrac{a+e}{2};\qquad \chi^2=\sum_i\frac{(O_i-E_i)^2}{E_i},\ \ E_i=N e_i
Jensen–Shannon is symmetric, bounded by ln 2 and finite with empty bins. Pearson's chi-square, with B − 1 degrees of freedom, and the L∞ distance max|aᵢ − eᵢ| are the usual choices for categorical features.

All of these statistics look at one feature at a time. They cannot see a change in how features move together: if x1x_1 and x2x_2 keep their histograms but their correlation flips sign, every per-feature PSI stays at its noise floor. The standard multivariate check is a domain classifier: label reference rows 0 and window rows 1, train a small classifier to tell them apart, and read its held-out AUC. An AUC near 0.5 means the two samples are indistinguishable; an AUC well above 0.5 means something moved, and the classifier's feature importances say where.

Professional Machine Learning Engineer exam guide, Google Cloud, Monitoring AI solutions: drift and skew

Maths, CS & AI Compendium, Henry Ndubuaku, Information theory; KL divergence

The rest of this lesson is in Pro

  • 04Window size: detection delay versus false alarms
  • 05Skew, label delay and monitoring predictions
  • 06Responding: alert, diagnose, retrain, roll out
  • 07Managed monitoring, and how the exams ask about it
  • Python challenge: Build a drift monitor
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between data drift and concept drift?
Data drift, or covariate shift, means the distribution of the inputs changes while the relationship between inputs and labels stays the same; it can be detected from unlabelled inputs. Concept drift means the relationship between inputs and labels changes; the inputs can look exactly as before, so it is only visible in labelled performance or label proxies. Prior shift, a change in class balance, is a third case that shows up first in the distribution of predictions.
What PSI value indicates drift?
The usual rule of thumb is below 0.1 for no meaningful change, 0.1 to 0.2 or 0.25 for a moderate change, and above that for a significant one. Treat these as starting points: with B bins, a window of N samples and a reference of M samples, PSI is about (B - 1)(1/N + 1/M) even when nothing changes, so small windows can cross 0.2 by chance. Check thresholds against a period you know was stable.
What is training-serving skew?
Training-serving skew is a difference between the features a model saw in training and the features it receives in production at the same time, caused by the system rather than the world: two implementations of a transformation, a different unit, a stale lookup or a missing join. It appears at deployment and does not go away by retraining. The fix is to compute features with shared code or a feature store for both paths and to monitor serving inputs against the training data.

Next in Production MLOps and Certification Prep

6.3Training pipelines, feature stores and servingReproducible pipelines, online/offline feature parity, and batch versus real-time serving architectures.

Sources

  • Professional Machine Learning Engineer exam guide, Google Cloud
  • Developing AI Apps and Agents on Azure (study guide), Microsoft
  • AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Hands-On Machine Learning, Aurélien Géron