1.3 Probability, Gaussians and maximum likelihood
A Gaussian is a mean and a covariance matrix: the covariance draws its ellipses, a Cholesky factor samples it, maximum likelihood recovers both from data, and slicing it at one coordinate produces a regression line, which is why least squares is Gaussian maximum likelihood.
01Random variables and densities
A random variable assigns a number to every outcome of an experiment. For a continuous variable, probabilities come from a density : the probability of landing in a region is the volume under the density over that region, and the total volume is 1.
The Gaussian (normal) density with mean and standard deviation is the bell in the widget below. The mean sets where the bell is centred; the standard deviation sets how wide it is. Drag the two cyan handles: the shaded area between them is the probability of landing there.
Worked example. Adult heights in some population have mean cm and standard deviation cm. A height of cm is standard deviations above the mean, and about of people are taller (half of the outside ). Subtracting the mean and dividing by maps any Gaussian onto the standard one, so a single table of answers every question about every Gaussian.
The viewport draws a two-dimensional Gaussian as a surface over the floor, with mean , , and correlation . With the surface factorises into two 1D bells, , and its peak is . Cyan points rain down from the same distribution and pile up where the surface is high.
A point at has : two standard deviations out along . Standardising every feature this way, with the mean and standard deviation of the training split, puts features on one scale before gradient descent, k-nearest neighbours or regularisation compare them.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 6.2 and 6.5
Deep Learning, Goodfellow, Bengio & Courville, Ch. 3.3 and 3.9.3
02The multivariate Gaussian: covariance is shape
Two definitions first. The expectation is the average value of over the distribution, its mean. The covariance is positive when and tend to be above their means together, negative when one tends to be above while the other is below, and is the variance . Dividing by gives the correlation .
In dimensions a Gaussian is set by a mean vector and a covariance matrix with entries . It is symmetric and positive definite: every direction has positive variance, for every , because is the variance of the projection . In 2D it is , the three numbers the cockpit exposes as , and .
- 1
Σ is symmetric, so it has orthonormal eigenvectors uᵢ (the columns of U) and real eigenvalues; positive definite means every λᵢ > 0.
The viewport is at , , , so with eigenvalues and . The major axis is tilted above the axis with half-length ; the minor half-length is . The peak rose from to because is times its uncorrelated value: the same unit of probability now sits on a narrower ridge.
Drag . The rings shear into a tilted ellipse and the arrows rotate, while the spread of each coordinate on its own stays and . The back walls show this: the emerald curves are the marginal densities and , the bell you get by ignoring the other coordinate, and the cyan bars are histograms of the landed samples' coordinates. The marginal variances set the bounding box; the correlation sets the shape inside it.
Flip the correlation to .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 4.4 and 6.5
Probabilistic Machine Learning: An Introduction, Kevin P. Murphy, Sec. 3.2.1 and 3.2.2
03Sampling with the Cholesky factor
A random number generator gives uniforms, and the Box-Muller transform turns two uniforms into two independent standard normals . To give them a covariance, factor with lower-triangular, the Cholesky factor, and map every through it:
Read the rows: uses the first normal only, and borrows from it. That shared is the correlation. The viewport rains 2,000 samples, each landing at for a fixed, seeded stream of , so dragging or slides the whole cloud instead of redrawing it. Hover to see its two columns as arrows.
One sample per row. The whole batch is X = mu + Z @ L.T: an (n, d) by (d, d) product, no Python loop. μ broadcasts across rows.
The same factor evaluates the density. Solve by forward substitution; then and , so neither nor is ever formed:
Undo the Cholesky map: slide Whiten to 1, so every sample moves from to .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 4.3 and 6.5.4
The rest of this lesson is in Pro
- 04Maximum likelihood: fitting μ and Σ
- 05Conditioning: slicing the density
- 06Least squares is Gaussian maximum likelihood
- 07Z-scores, Mahalanobis outliers and normality
- Python challenge: Log-density, MLE and conditioning
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between Mahalanobis distance and Euclidean distance?
- Euclidean distance treats every direction alike. Mahalanobis distance first rescales by the covariance matrix, so it measures distance in standard deviations along each principal axis and accounts for correlations between features. Points at the same Mahalanobis distance from the mean lie on the same density contour of a Gaussian, which makes it the natural score for multivariate outliers.
- Why does the maximum likelihood estimate of variance divide by n instead of n minus 1?
- Maximising the Gaussian likelihood over the mean and the variance gives the average squared deviation from the sample mean, which divides by n. Because the sample mean is fitted to the same data, that estimate is biased low by a factor of (n − 1)/n. Dividing by n − 1 gives the unbiased sample variance. The two differ noticeably only for small samples.
- Why is least squares the same as maximum likelihood?
- If each target equals a linear prediction plus independent Gaussian noise with constant variance, the log-likelihood of the data is a constant minus the sum of squared errors divided by twice the noise variance. Maximising it over the weights is therefore the same as minimising the sum of squared errors. Under other noise models the equivalence breaks: Laplace noise leads to absolute error instead.
Next in Mathematical Foundations
1.4Convexity, Lagrange multipliers and dualityWhen a minimum is guaranteed to be global, and how constraints reshape the landscape — the math behind SVMs and regularisation.Sources
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
- Maths, CS & AI Compendium, Henry Ndubuaku
- Deep Learning, Goodfellow, Bengio & Courville
- Probabilistic Machine Learning: An Introduction, Kevin P. Murphy