Skip to content
LearnMLByDoing
Free

0.3 Matrices: tables of numbers that move space

30 minFree lessonbeginner3B1BMMLKhanDLStrang
Builds on0.2Vectors

A 2×2 matrix is four numbers that tell every point of the plane where to move, and once you can watch that movement, multiplying matrices and vectors becomes a picture you can check by eye.

01A matrix is a table that moves the plane

Look at the viewport. The faint square grid is the plane before anything happens. The brighter grid is the same plane after a matrix has moved it. While the animation plays, the plane glides from its starting position to the moved one and back. Notice what never changes: grid lines stay straight, they stay evenly spaced, and the origin stays where it is.

A matrix is written as a small table of numbers. The one in the viewport is this:

A=[1101]\tok{mat_a}{A}=\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}
A shear: the top of the letter F slides to the right while its foot stays put.

Here is the key idea of the whole lesson: read the table by its columns. Column 1, (1,0)(1, 0), is where the emerald arrow ı^\tok{ihat}{\hat\imath} lands; ı^\hat\imath starts as one step along xx. Column 2, (1,1)(1, 1), is where the amber arrow ȷ^\tok{jhat}{\hat\jmath} lands; it starts as one step up yy. In the viewport ı^\hat\imath stays where it was, and ȷ^\hat\jmath tips over to (1,1)(1, 1), taking every vertical grid line with it.

Why do two arrows decide everything? Every point is an amount of ı^\hat\imath plus an amount of ȷ^\hat\jmath: the cyan point (2,1)(2, 1) is two steps of ı^\hat\imath and one of ȷ^\hat\jmath. The matrix keeps those amounts and swaps in the moved arrows, so (2,1)(2, 1) lands on 2⋅(1,0)+1⋅(1,1)=(3,1)2\cdot(1, 0) + 1\cdot(1, 1) = (3, 1). Know where the two arrows go, and you know where every point goes. The F inside the emerald unit square is carried along in the same way.

Swap the shear for a quarter turn: column 1 becomes (0,1)(0, 1) and column 2 becomes (−1,0)(-1, 0).

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3, Linear transformations and matrices

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.2 and 2.7

02Matrix times vector: a weighted sum of columns

The viewport now uses a new matrix. The faint cyan arrow is the vector v=(1,2)\tok{vec_v}{v} = (1, 2) and the bright cyan arrow is where the matrix sends it, written Av\tok{image}{Av}. The two thin arrows show the route: one step along the emerald column, then two steps along the amber column.

A=[2−111],v=[12]\tok{mat_a}{A}=\begin{bmatrix}2 & -1\\ 1 & 1\end{bmatrix},\qquad \tok{vec_v}{v}=\begin{bmatrix}1\\ 2\end{bmatrix}
Two ways to compute Av, one answer1/3
  1. 1
    Av=1[21]+2[−11]\tok{image}{Av} = 1\begin{bmatrix}2\\ 1\end{bmatrix} + 2\begin{bmatrix}-1\\ 1\end{bmatrix}

    Column picture: the entries of v say how much of each column to take, 1 of column 1 and 2 of column 2.

Loading figure
Change any entry with the small arrows, then step through row 1, row 2 and the column view. Your edits also move the 3D viewport.

Change the probe to v=(1,−1)v = (1, -1): one step along column 1, then one step backwards along column 2.

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Linear algebra: matrix-vector products

03Stretch, rotate, shear and flip

A few basic moves cover most of what a 2×2 matrix can do, and the preset menu in the cockpit has one of each. The viewport starts on a stretch: column 1 is (2,0)(2, 0), so ı^\hat\imath doubles in length, while ȷ^\hat\jmath stays at (0,1)(0, 1). The grid is pulled sideways, the unit square becomes a 2-by-1 rectangle, and the probe v=(1,1)v = (1, 1) lands on (2,1)(2, 1).

[2001]⏟stretch[0−110]⏟quarter turn[1101]⏟shear[−1001]⏟mirror\underbrace{\begin{bmatrix}2 & 0\\ 0 & 1\end{bmatrix}}_{\text{stretch}}\quad \underbrace{\begin{bmatrix}0 & -1\\ 1 & 0\end{bmatrix}}_{\text{quarter turn}}\quad \underbrace{\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}}_{\text{shear}}\quad \underbrace{\begin{bmatrix}-1 & 0\\ 0 & 1\end{bmatrix}}_{\text{mirror}}
Read each one by its columns: first where î goes, then where ĵ goes.

You can predict every picture from the columns alone. In the quarter turn, ı^\hat\imath goes to (0,1)(0, 1) and ȷ^\hat\jmath goes to (−1,0)(-1, 0): both arrows turn by 90 degrees. In the mirror, ı^\hat\imath goes to (−1,0)(-1, 0) while ȷ^\hat\jmath stays, so left and right swap over.

Apply the mirror: ı^\hat\imath lands on (−1,0)(-1, 0) and ȷ^\hat\jmath stays at (0,1)(0, 1).

Freeze a shear halfway: pause the animation at progress t=0.5t = 0.5.

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3

Introduction to Linear Algebra, Gilbert Strang, Sec. 8.1

04The determinant: how much area grows

Watch the emerald unit square. It starts with area 1, and the matrix turns it into a slanted box called a parallelogram. In the viewport its sides are the two columns, (2,0)(2, 0) and (1,1.5)(1, 1.5): a base of 2 and a height of 1.5, so its area is 2×1.5=32 \times 1.5 = 3. Every grid cell grows by the same factor of 3, because the moved grid stays evenly spaced.

det⁡A=ad−bc=2⋅1.5−1⋅0=3\tok{det}{\det A} = ad - bc = 2\cdot 1.5 - 1\cdot 0 = 3
For A = [[a, b], [c, d]] = [[2, 1], [0, 1.5]]. The 'det A' readout in the viewport shows the same number.
Loading figure
Drag the tips of the two columns. The determinant is the area of the moved square, with a sign; the 3D viewport follows your drag.

The sign carries information too. A negative determinant means the plane has been flipped over: the F reads backwards and the square turns rose. A determinant of zero means the square has been flattened onto a line, and points that started in different places now land on top of each other.

Flip it: keep column 1 at (2,0)(2, 0) and send ȷ^\hat\jmath below it, to (1,−1.5)(1, -1.5).

Squash it: send ȷ^\hat\jmath to (1,0.5)(1, 0.5), exactly half of column 1, which is now (2,1)(2, 1).

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 6, The determinant

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 4.1

05Matrix times matrix: one move after another

Do a quarter turn, then a shear. The result is again one move of the plane that keeps lines straight and the origin fixed, so it is again a matrix. The viewport shows that combined matrix: watch ı^\hat\imath head to (1,1)(1, 1) and ȷ^\hat\jmath to (−1,0)(-1, 0).

[1101]⏟shear S  [0−110]⏟turn R=[1−110]\underbrace{\begin{bmatrix}1 & 1\\ 0 & 1\end{bmatrix}}_{\text{shear } S}\;\underbrace{\begin{bmatrix}0 & -1\\ 1 & 0\end{bmatrix}}_{\text{turn } R} = \begin{bmatrix}1 & -1\\ 1 & 0\end{bmatrix}
SR: the turn R acts first, then the shear S. A product is read right to left.
Each column of SR is S applied to a column of R1/4
  1. 1
    R ı^=[01],R ȷ^=[−10]R\,\hat\imath = \begin{bmatrix}0\\ 1\end{bmatrix},\qquad R\,\hat\jmath = \begin{bmatrix}-1\\ 0\end{bmatrix}

    The columns of R: where the turn sends the two arrows.

Loading figure
Pick two moves A and B and press play. The left panel does B and then A; the right panel does A and then B.

Do the same two moves in the other order, shear first and then turn: RS=[0−111]RS = \begin{bmatrix}0 & -1\\ 1 & 1\end{bmatrix}.

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 4, Matrix multiplication as composition

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.2.1

06A neural network layer: matrix times input, plus a bias

A layer of a neural network takes a list of input numbers and produces a list of output numbers. Its main ingredient is the move you now know: multiply the input by a weight matrix W\tok{mat_a}{W}. Then it adds a bias vector b\tok{bias}{b}, which slides the result. In the viewport the matrix is the one from step 2, and the bias b=(1,−1)b = (1, -1) shifts the grid, the square and the F as one piece; the grey arrow shows where the origin slides to.

y=Wx+b=[2−111][11]+[1−1]=[12]+[1−1]=[21]y = \tok{mat_a}{W}\tok{vec_v}{x} + \tok{bias}{b} = \begin{bmatrix}2 & -1\\ 1 & 1\end{bmatrix}\begin{bmatrix}1\\ 1\end{bmatrix} + \begin{bmatrix}1\\ -1\end{bmatrix} = \begin{bmatrix}1\\ 2\end{bmatrix} + \begin{bmatrix}1\\ -1\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}
The faint cyan arrow is the input x = (1, 1); the bright one ends at the layer's output y = (2, 1), the 'Av + b' readout.
W (weight matrix)[,]4 values

Here 2 inputs and 2 outputs. A layer that reads a 28×28 image (784 inputs) into 128 outputs has a 128 × 784 weight matrix: 100,352 numbers, each one a knob that training adjusts.

Once a bias is added the origin moves, so Wx+bWx + b on its own is a matrix move followed by a slide; grid lines still stay straight and evenly spaced. Real networks then pass every output through a small bending function, called an activation, so that a stack of layers can do far more than one matrix could.

Move the bias to b=(−1,1)b = (-1, 1).

Deep Learning, Goodfellow, Bengio & Courville, Ch. 2.1 and 2.2

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.2

Compiling scene
Loading editor

Questions learners ask

What is a matrix, and why does machine learning use so many of them?
A matrix is a rectangular table of numbers. Geometrically, a 2 by 2 matrix moves every point of the plane: its first column is where the arrow (1, 0) lands and its second column is where (0, 1) lands. Machine learning uses matrices because a layer of a neural network is a matrix multiplication plus a bias, and because a whole dataset can be stored as a matrix with one row per example, so a single multiplication processes all of it at once.
How do you multiply a matrix by a vector?
Take each entry of the vector as an amount of the matching column, then add the scaled columns: with columns (2, 1) and (-1, 1) and the vector (1, 2), the answer is 1 times (2, 1) plus 2 times (-1, 1), which is (0, 3). The row method gives the same numbers: each output entry is one row of the matrix multiplied entry by entry with the vector and added up. In NumPy you write A @ v.
Why does the order of matrix multiplication matter?
Because each matrix is a move of space, and the product AB means do B first and then A. Turning a shape and then shearing it usually gives a different result from shearing first and then turning, so AB and BA are usually different matrices. Their determinants always match, though, because the area factors multiply in either order.

Next in Maths Primer

0.4Derivatives: slopes, tangents and the chain ruleZoom into a curve until it looks straight: that slope is the derivative. Then chain functions together and watch rates of change multiply.

Sources

  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy
  • Deep Learning, Goodfellow, Bengio & Courville
  • Introduction to Linear Algebra, Gilbert Strang