Skip to content
LearnMLByDoing
Free

2.3 Bias, variance and regularisation

28 minProintermediateISLRHOMLDLESLMLCC

Expected test error splits into bias², variance and irreducible noise, and a penalty on the coefficients buys a large cut in variance for a small rise in bias, with λ chosen by cross-validation rather than by the training error.

01Expected test error has three parts

A model is fitted to one training set, but we care about its error on new data, and a different training set would have given a different model. The viewport makes that concrete. The truth is f(x)=sin⁡(2.6x+0.6)f(x)=\sin(2.6x+0.6), the white curve. Twenty inputs sit at x=−0.95,−0.85,…,0.95x = -0.95, -0.85, \dots, 0.95, and each of 30 training sets draws fresh labels y=f(x)+εy = f(x) + \varepsilon with noise of standard deviation σ=0.3\sigma = 0.3. Every training set gets its own cubic fit f^\tok{fhat}{\hat f}, which enters at the front of the stack with its 20 cyan points and pushes the earlier fits back.

The intuition first, before any formula. Train the same model on 30 slightly different datasets. If the 30 answers disagree with each other, that is variance. If they agree with each other but are all wrong in the same way, that is bias. A dartboard shows both at once: each dart is one trained model, the bullseye is the truth. Try the four presets, then turn on the averaging toggle.

Loading figure
The same colours as the viewport: amber for bias, rose for variance. Averaging several throws per dart is what bagging does with models (lesson 2.4): the variance shrinks, the bias stays.

Once all 30 are drawn, the stack collapses onto the front plane and the fits contract onto their average fˉ(x)=ED[f^(x)]\tok{fbar}{\bar f}(x) = \mathbb{E}_D[\hat f(x)], the bold emerald curve, leaving the amber gap to the truth. The spread you saw before the contraction is variance; the gap that remains is bias.

One input, real numbers. The amber-and-rose marker on the dashed vertical line is a probe at x0=0.6x_0 = 0.6. The emerald beads are where each of the 30 cubic fits crosses that line: their values run from 0.6450.645 to 1.1891.189. Their mean is fˉ(0.6)=0.954\tok{fbar}{\bar f}(0.6) = 0.954 (emerald dot) and the truth is f(0.6)=0.831f(0.6) = 0.831 (white dot). So at this input the squared bias is (0.954−0.831)2=0.0151(0.954 - 0.831)^2 = 0.0151, the amber segment, and the variance, the average squared distance of the 30 values from 0.9540.954, is 0.01590.0159, the rose bracket of ±1\pm 1 standard deviation (0.1260.126). Drag Probe input x₀ in the cockpit; the HUD reads both numbers at every x0x_0.

E[(y−f^)2]=(fˉ−f)2⏟Bias2+ED[(f^−fˉ)2]⏟Var+σ2\mathbb{E}\big[(y-\tok{fhat}{\hat f})^2\big]=\underbrace{(\tok{fbar}{\bar f}-f)^2}_{\tok{bias}{\mathrm{Bias}^2}}+\underbrace{\mathbb{E}_D\big[(\tok{fhat}{\hat f}-\tok{fbar}{\bar f})^2\big]}_{\tok{variance}{\mathrm{Var}}}+\tok{noise}{\sigma^2}
Every function is evaluated at one fixed input x. The expectation runs over training sets D and over the noise in the new label y.
Where the three terms come from1/4
  1. 1
    y−f^(x)=ε+(f(x)−fˉ(x))+(fˉ(x)−f^(x))y-\hat f(x)=\varepsilon+\big(f(x)-\bar f(x)\big)+\big(\bar f(x)-\hat f(x)\big)

    Add and subtract f(x) and the average fit. ε is the new label's noise: mean 0, variance σ², independent of the training set.

f̂ on the evaluation grid[,]3,630 values

One row per fitted model. f̄ is the column mean; bias² averages (f̄ − f)² over the columns, variance averages each column's variance over the rows.

The inset at the bottom right splits the bar exactly this way. At degree 3 it reads σ2=0.090\sigma^2 = 0.090, bias² =0.0061= 0.0061 and variance =0.0177= 0.0177, so the expected test MSE is 0.11380.1138. The cyan tick marks the average training MSE, 0.07360.0736.

An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 2.2.2, The Bias-Variance Trade-Off

The Elements of Statistical Learning, Hastie, Tibshirani & Friedman, Ch. 7.3

02Capacity: underfitting and overfitting

The degree d\tok{degree}{d} sets the model's capacity: the fit uses x,x2,…,xdx, x^2, \dots, x^d plus an intercept, so it has d+1d + 1 parameters. The lower chart in the inset plots the four quantities against dd on a log scale: test MSE in white, training MSE in cyan, bias² in amber and variance in rose. The emerald line marks the current degree.

Underfitting. At d=1d = 1 a straight line cannot follow the sine: bias² =0.1870= 0.1870 dominates, variance is only 0.00760.0076, and training and test MSE nearly agree at 0.28330.2833 and 0.28460.2846. Overfitting. The viewport is now at d=15d = 15: 16 parameters for 20 points. The fits chase the noise of their own training set, the stack becomes a tangle of curves that leave the plot near the edges, and variance reaches 1.3321.332 while bias² is only 0.01040.0104. Test MSE is 1.4321.432, more than twelve times its value at d=3d = 3, while training MSE falls to 0.03150.0315.

Drop to a straight line, d=1d = 1, and move the probe to the right edge, x0=0.95x_0 = 0.95.

Walk up the degree: d=8d = 8, no penalty.

Keep climbing: d=12d = 12.

train MSE⏟falls with dvs.Bias2+Var+σ2⏟U-shaped in d\underbrace{\text{train MSE}}_{\text{falls with }\tok{degree}{d}}\qquad\text{vs.}\qquad \underbrace{\tok{bias}{\mathrm{Bias}^2}+\tok{variance}{\mathrm{Var}}+\tok{noise}{\sigma^2}}_{\text{U-shaped in }\tok{degree}{d}}
Nested models: a degree-d polynomial can always reproduce any lower-degree fit, so training error can only fall.

Drag dd from 1 to 15 and read the chart: bias² falls quickly and then flattens, variance grows slowly and then explodes, and their sum with σ2\sigma^2 is lowest at d=3d = 3 (0.1138), with d=4d = 4 a hair behind at 0.1140.

At d=15d = 15, put the probe between the last two training inputs, x0=0.9x_0 = 0.9.

Footnote: double descent. The slider stops at d=15d = 15, below n=20n = 20, the classical regime where this U-shape holds. When the number of parameters exceeds nn, least squares has infinitely many interpolating solutions, and the minimum-norm one, which gradient descent from zero converges to, can see test error fall again as capacity keeps growing (Belkin et al., 2019). The peak sits at the interpolation threshold, where the fit is forced through every noisy point; a well-tuned ridge penalty largely removes it.

An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 2.2.2 and 7.1

Hands-On Machine Learning, Aurélien Géron, Ch. 4, Polynomial Regression

Machine Learning Crash Course, Google, Overfitting

03Ridge regression: penalise the squared norm

Instead of lowering the degree, keep all 15 powers and make large coefficients expensive. Ridge regression adds λ\lambda times the squared norm of the weights to the residual sum of squares. The powers are standardised first, so the penalty treats every coefficient alike, and the intercept bb is left out of it:

min⁡w, b ∑i=1n(yi−b−xi⊤w)2+λ ∥w∥22\min_{\tok{coef}{w},\,b}\ \sum_{i=1}^{n}\big(y_i-b-x_i^{\top}\tok{coef}{w}\big)^2+\tok{lambda}{\lambda}\,\|\tok{coef}{w}\|_2^2
Same notation as lesson 2.2: X holds the features (here the standardised powers), w the slopes, b the intercept.
The closed form1/4
  1. 1
    b^=yˉ−xˉ⊤w ⇒ min⁡w ∥yc−Xcw∥2+λ∥w∥2\hat b=\bar y-\bar x^{\top}w\ \Rightarrow\ \min_w\ \|y_c-X_cw\|^2+\lambda\|w\|^2

    Setting the derivative in b to zero gives b̂. Substituting it centres every column and the targets: X_c = X − 1x̄ᵀ, y_c = y − ȳ. Centring removes the intercept from the problem, so the penalty cannot touch it.

The viewport is back at d=15d = 15, now with λ=0.1\lambda = 0.1. The tangle calms into a tight bundle: variance drops from 1.3321.332 to 0.04610.0461, bias² is 0.00260.0026, and the expected test MSE falls from 1.4321.432 to 0.13870.1387, close to the best low-degree fit. The training MSE rises a little, from 0.03150.0315 to 0.05690.0569: the fits no longer chase every point.

Ridge spends its flexibility only where the data support it. Hover λ\tok{lambda}{\lambda} to read the effective degrees of freedom df(λ)\mathrm{df}(\lambda), defined in the next step: 11.84 at λ=10−6\lambda = 10^{-6}, 6.20 at λ=0.1\lambda = 0.1 and 4.69 at λ=1\lambda = 1, for a model that nominally has 15 slopes. Push λ\lambda to 10 and the penalty wins: bias² climbs to 0.09820.0982 and test MSE to 0.20420.2042. Too much regularisation is underfitting by another name.

Raise the penalty to λ=1\lambda = 1 (log⁡10λ=0\log_{10}\lambda = 0) and put the probe at x0=0.9x_0 = 0.9, where the unpenalised fits scattered most.

An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 6.2.1, Ridge Regression

The Elements of Statistical Learning, Hastie, Tibshirani & Friedman, Ch. 3.4.1

Hands-On Machine Learning, Aurélien Géron, Ch. 4, Ridge Regression

The rest of this lesson is in Pro

  • 04Shrinkage along singular directions
  • 05Lasso: exact zeros from the L1 corner
  • 06Choosing λ with k-fold cross-validation
  • 07Regularisation in practice
  • Python challenge: Ridge, soft thresholding and leak-free cross-validation
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the bias-variance tradeoff?
The expected squared error of a model on new data splits into three parts: the squared bias, how far the average model over many training sets is from the truth; the variance, how much the model changes from one training set to another; and the irreducible noise in the labels. Flexible models have low bias but high variance, rigid models the reverse, and the best test error sits in between. Regularisation and more data reduce variance; more capacity or better features reduce bias.
What is the difference between ridge and lasso regression?
Both add a penalty on the size of the coefficients to the squared-error loss. Ridge penalises the sum of squared coefficients, has a closed-form solution and shrinks every coefficient smoothly without making any of them exactly zero. Lasso penalises the sum of absolute values, has no closed form in general, and sets some coefficients exactly to zero, so it also performs feature selection. With strongly correlated features the lasso picks one somewhat arbitrarily, which the elastic net, a mix of both penalties, fixes.
How do I choose the regularization parameter lambda?
Use k-fold cross-validation on the training data: for each value on a logarithmic grid, fit on k minus 1 folds, measure the error on the held-out fold, and average. Pick the value with the lowest average, or the largest value within one standard error of it for a simpler model. Fit every preprocessing step, such as scaling, inside each fold, and keep a separate test set to report the final error.

Next in Classical Machine Learning

2.4Decision trees, random forests and gradient boostingAxis-aligned partitions of feature space, bagging to cut variance, and boosting as gradient descent in function space.

Sources

  • An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
  • Hands-On Machine Learning, Aurélien Géron
  • Deep Learning, Goodfellow, Bengio & Courville
  • The Elements of Statistical Learning, Hastie, Tibshirani & Friedman
  • Machine Learning Crash Course, Google