1.1 Linear transformations, eigenvectors and SVD in 3D
Every 3×3 matrix turns the unit sphere into an ellipsoid, and its eigenvectors and singular vectors tell you exactly how.
01A matrix is a function on space
A matrix is a rule that moves every point to . Because the rule is linear, three outputs fix it completely: the places where the standard basis vectors land.
Worked example. The viewport's matrix has columns , and . Take : one step along and two along . Then . Row by row gives the same answer: the first output is .
Those three images are the columns of . In the viewport they are the cyan, emerald and amber arrows. Every other point, including the 500 samples on the unit sphere, keeps the coordinates it had relative to , now measured along the new arrows.
Lines stay lines and the origin stays fixed, so the unit sphere always lands on an ellipsoid, possibly flattened. That one shape summarises the whole matrix. The rest of the lesson reads it three ways: by volume, by invariant directions, and by its principal axes.
Row i produces output coordinate i; column j is the image of e_j.
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3, Linear transformations and matrices
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.7
02Volume, orientation and rank
The unit cube spanned by maps to the parallelepiped spanned by the columns of . Its signed volume is the determinant. Any region can be tiled by small cubes, and each cube is scaled by the same factor, so multiplies every volume by .
The sign records orientation. A negative determinant means the map contains a reflection: a right-handed frame comes out left-handed. A zero determinant means the columns are linearly dependent. Space is squashed onto a plane or a line, and the ellipsoid becomes a flat disc or a segment. The dimension of that image is the rank.
- 1
The third column is the sum of the first two: Ae₃ = Ae₁ + Ae₂.
Lose two dimensions: make column 2 twice column 1 and set column 3 to zero.
Introduction to Linear Algebra, Gilbert Strang, Ch. 5, Determinants
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 4.1
03Eigenvectors: directions that only stretch
Most vectors are turned by . An eigenvector is a nonzero direction that comes out on its own line, scaled by its eigenvalue:
Watch it happen in 2D first. The widget sends a test vector (white) round the unit circle and draws its image (amber). Most of the time the two point in different directions. Twice per half-turn they line up, and an emerald tick marks the direction: that is an eigenvector, and at that moment is its eigenvalue.
Worked example (2×2). For the widget's matrix, . A nonzero with exists only if this matrix squashes the plane, that is, if its determinant is zero: . So , which gives and . Check: and . The 3×3 recipe below is the same, with a cubic instead of a quadratic.
- 1
Move both terms to one side.
A real cubic always has at least one real root, so every real matrix has at least one invariant line. The other two roots may form a complex-conjugate pair. Then rotates a plane, and no other real direction survives. The viewport draws only the real eigenvectors: emerald lines through the origin, with the bright segment reaching .
Replace the two off-diagonal entries and by their average, and , so that .
Turn the x–y plane instead of stretching it: .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 14, Eigenvectors and eigenvalues
Introduction to Linear Algebra, Gilbert Strang, Sec. 6.1
04Symmetric matrices: orthogonal eigenvectors
When the picture becomes clean. The spectral theorem guarantees real eigenvalues and an orthonormal basis of eigenvectors, so the matrix factors as
Read the product right to left. rotates the eigenvectors onto the coordinate axes, stretches each axis by its eigenvalue, and rotates back. In the viewport, the emerald eigenvector lines now run exactly along the ellipsoid's axes, and the semi-axis lengths equal .
A symmetric matrix with every is called positive semidefinite (PSD), and positive definite when every . Then for every : the function is a bowl that curves up in every direction. The viewport's matrix has , all positive. A covariance matrix is always PSD (lesson 1.3), and a Hessian that is PSD everywhere means a convex loss (lesson 1.4).
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 4.4
Introduction to Linear Algebra, Gilbert Strang, Sec. 6.4
05SVD: rotate, stretch, rotate
A general matrix has no orthogonal eigenbasis. There is always, however, an orthonormal input frame that maps onto an orthogonal output frame. That is the singular value decomposition, and every matrix has one:
- 1
AᵀA is symmetric, and xᵀAᵀAx = ‖Ax‖² ≥ 0, so it is positive semidefinite: its eigenvalues are real and not negative. The spectral theorem from the previous step applies, and V holds its orthonormal eigenvectors.
Scrub the SVD stage slider. From 0 to 1, rotates the sphere: the surface looks unchanged, but the coloured sample points move. From 1 to 2, stretches along the coordinate axes, so the ellipsoid is axis-aligned at stage 2. From 2 to 3, rotates it into place, and the cyan bars become its semi-axes.
Load a matrix with and stop at stage 1, then drag the SVD stage slider slowly from 1 to 2.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 4.5
Introduction to Linear Algebra, Gilbert Strang, Ch. 7
06Power iteration and PCA
You rarely need every eigenvector. Power iteration finds the dominant one using only matrix-vector products. The viewport is running it now: the amber arrow starts at , its image grows out of it as a faint arrow (it lands on the ellipsoid), and then turns onto that direction, shrunk back to length 1. The amber dots are the earlier iterates.
Write the start vector in the eigenbasis, . After steps the component along carries a factor , so everything except shrinks relative to it like , provided . A large eigengap gives fast convergence. Two eigenvalues of equal magnitude give none.
In the viewport the matrix is the symmetrised default, with , so . Each step roughly halves the angle to the emerald : the HUD shows falling from to under half a degree at , while the Rayleigh quotient climbs to . Then the loop holds and starts again.
Give the two largest eigenvalues almost the same size: .
Now separate them: .
Principal component analysis applies the same idea to data. Centre the data matrix to get (subtract each column's mean), form the covariance , and keep the eigenvectors with the largest eigenvalues. Equivalently, take the top right singular vectors of . Projecting onto the top of them keeps more variance than any other -dimensional subspace.
Switch off the power iteration and show the reference data cloud the challenge uses, with its principal components.
One sample per row. Subtract the column means before anything else.
Orthonormal rows, largest variance first.
Coordinates of each sample in the principal basis.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 10.2
Introduction to Linear Algebra, Gilbert Strang, Sec. 7.3
07Low-rank approximation: keep the big singular values
The SVD writes any matrix as a sum of simple pieces, one per singular value. Each piece is a rank-1 matrix: a column times a row. Keeping only the largest pieces gives a cheaper matrix , and no other rank- matrix is closer to .
The viewport keeps of the default matrix's three singular values, . The ellipsoid has flattened into a disc along its shortest axis, and the HUD reports the error . Move the Keep top-k slider to 1 and the disc shrinks to a segment, with error .
For data, the same bookkeeping is the explained variance ratio. The PCA eigenvalues are variances, and is the share of the total variance that component keeps. Plotting these shares in decreasing order gives a scree plot. For the challenge's cloud the shares are about , and , so two components keep about .
Keep only the largest singular value: .
Introduction to Linear Algebra, Gilbert Strang, Sec. 7.1
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 4.6
Questions learners ask
- What is the difference between eigenvectors and singular vectors?
- An eigenvector v satisfies Av = λv, so it keeps its own direction. It exists only for square matrices and need not form an orthogonal set. Singular vectors come in pairs: A maps the right singular vector vᵢ to σᵢuᵢ, where uᵢ is the left singular vector. Both sets are orthonormal and exist for any matrix, including rectangular ones. For a symmetric positive semidefinite matrix the two notions coincide.
- Why does PCA use the eigenvectors of the covariance matrix?
- The variance of the centred data along a unit direction w is wᵀCw, where C is the covariance matrix. Maximising this quadratic form over unit vectors gives the top eigenvector of C, and repeating under orthogonality constraints gives the next ones. The same directions are the right singular vectors of the centred data matrix, which is how most libraries compute them.
- When does power iteration fail or converge slowly?
- The error shrinks like |λ₂/λ₁|ᵏ, so convergence is slow when the two largest eigenvalues are close in magnitude. It fails when they have equal magnitude, for example λ and −λ, or a complex pair. It also fails if the start vector has no component along the dominant eigenvector, although rounding errors usually introduce one. Shifts, deflation, or Krylov methods such as Lanczos fix these cases.
Next in Mathematical Foundations
1.2Gradients, Jacobians and the chain ruleLocal linear approximations of functions between vector spaces, and why backprop is just the chain rule applied in order.Sources
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Introduction to Linear Algebra, Gilbert Strang
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)