Skip to content
LearnMLByDoing
Free

0.2 Vectors: arrows, adding, scaling and the dot product

30 minFree lessonbeginner3B1BMMLKhan

A vector is an arrow and a list of numbers at the same time, and the dot product turns two of them into one number that says how much they point the same way.

01A vector is an arrow and a list

Watch the cyan dot in the viewport: it walks 3 squares right, then 1 square up, and then the emerald arrow shoots from the centre (the origin) straight to where the dot stopped. That arrow is a vector: something with a direction and a length. Where it starts does not matter; slide it anywhere and it is the same vector.

The two moves are all you need to rebuild the arrow, so we can also write the vector as the list of moves: a=(3, 1)\tok{vec_a}{\mathbf{a}} = (3,\ 1). The first number is the step along xx (right is positive, left is negative), the second the step along yy (up is positive). The label at the tip shows both views at once.

a=(a1, a2)=(3, 1)\tok{vec_a}{\mathbf{a}} = (a_1,\ a_2) = (3,\ 1)
Arrow and list are two pictures of one object.

Give the first number a minus sign: a=(−2, 3)\mathbf{a} = (-2,\ 3).

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 1, Vectors

02Length: Pythagoras on the two moves

The two moves and the arrow make a right-angled triangle: 3 along, 4 up, and the arrow is the long side. How long is it? Hover ∣a∣\tok{length}{|\mathbf{a}|} to draw a circle of that radius around the origin: every arrow that ends on the circle has the same length.

Pythagoras says the long side, squared, equals the two short sides squared and added: 32+42=9+16=253^2 + 4^2 = 9 + 16 = 25. The length is the number that gives 25 when multiplied by itself, 25=5\sqrt{25} = 5, and the tip label reads ∣a∣=5|\mathbf{a}| = 5.

∣a∣=a12+a22=32+42=25=5\tok{length}{|\mathbf{a}|} = \sqrt{a_1^2 + a_2^2} = \sqrt{3^2 + 4^2} = \sqrt{25} = 5

Go back to a=(3, 1)\mathbf{a} = (3,\ 1).

Flip both signs: a=(−3, −4)\mathbf{a} = (-3,\ -4).

03Adding: one trip, then the other

Adding two vectors means doing one trip and then the other. In the viewport the amber arrow b\tok{vec_b}{\mathbf{b}} slides until its tail sits on the tip of a\tok{vec_a}{\mathbf{a}}, and the cyan arrow from the origin to where you end up is the sum a+b\tok{sum}{\mathbf{a}+\mathbf{b}}. This is called adding tip to tail.

Because each vector is a list of moves, you add the moves: right 3+1=43 + 1 = 4, up 1+2=31 + 2 = 3, so (3, 1)+(1, 2)=(4, 3)(3,\ 1) + (1,\ 2) = (4,\ 3). Its length is 16+9=5\sqrt{16 + 9} = 5, a little less than ∣a∣+∣b∣≈3.16+2.24=5.40|\mathbf{a}| + |\mathbf{b}| \approx 3.16 + 2.24 = 5.40, because the route bends.

a+b=(a1+b1, a2+b2)=(3+1, 1+2)=(4, 3)\tok{sum}{\mathbf{a}+\mathbf{b}} = (a_1 + b_1,\ a_2 + b_2) = (3 + 1,\ 1 + 2) = (4,\ 3)

After the sum appears, a faint emerald line traces the other order: b\mathbf{b} first, then a\mathbf{a}. It lands on the same point, so the order of adding does not matter, and the two routes outline a parallelogram.

Make b=(−3, −1)\mathbf{b} = (-3,\ -1), the exact reverse of a\mathbf{a}.

04Scaling: stretch, shrink and flip

Multiplying a vector by an ordinary number stretches it along its own line. In the viewport the cyan arrow grows from a\mathbf{a} to c a\tok{scale_c}{c}\,\mathbf{a} and back, while the faint emerald arrow keeps the original for comparison.

With a=(2, 1)\mathbf{a} = (2,\ 1) and c=2c = 2, every component doubles: 2×(2, 1)=(4, 2)2 \times (2,\ 1) = (4,\ 2). The direction stays the same and the length doubles, from 5≈2.24\sqrt{5} \approx 2.24 to 20≈4.47\sqrt{20} \approx 4.47. A cc between 0 and 1 shrinks instead: 0.5×(2, 1)=(1, 0.5)0.5 \times (2,\ 1) = (1,\ 0.5).

c a=(c a1, c a2)=2⋅(2, 1)=(4, 2)\tok{scale_c}{c}\,\tok{vec_a}{\mathbf{a}} = (c\,a_1,\ c\,a_2) = 2\cdot(2,\ 1) = (4,\ 2)

Set c=−1c = -1.

Subtraction is adding a flipped vector: a−b=a+(−1) b\mathbf{a} - \mathbf{b} = \mathbf{a} + (-1)\,\mathbf{b}. For (3, 1)−(1, 2)(3,\ 1) - (1,\ 2) that is (3−1, 1−2)=(2, −1)(3 - 1,\ 1 - 2) = (2,\ -1), the arrow from the tip of b\mathbf{b} to the tip of a\mathbf{a}.

05The dot product: multiply, then add

The dot product turns two arrows into one number that measures how much they point the same way. The arc between them in the viewport is coloured by it: emerald when the arrows agree, grey when they are at a right angle, rose when they point against each other.

The recipe uses the lists: multiply matching components, then add the results. For a=(3, 1)\mathbf{a} = (3,\ 1) and b=(1, 3)\mathbf{b} = (1,\ 3) that is 3×1+1×3=3+3=63\times1 + 1\times3 = 3 + 3 = 6. The widget lights up each product in turn, and its buttons move the arrows in the viewport.

Loading figure
Change a component and watch the products, the total and the arrows. Try making one product negative.
a⋅b=a1b1+a2b2=3⋅1+1⋅3=6\tok{dot}{\mathbf{a}\cdot\mathbf{b}} = a_1 b_1 + a_2 b_2 = 3\cdot 1 + 1\cdot 3 = 6

Let b\mathbf{b} turn around the origin while a\mathbf{a} stays put.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 3.2, Inner products

06Angle and shadow: why it measures agreement

Shine a light straight down onto the line through b\mathbf{b} (the faint amber line): a\mathbf{a} casts a shadow, the grey bar, and the dashed drop meets the line at a right angle. The more a\mathbf{a} leans toward b\mathbf{b}, the longer the shadow. The dot product is the shadow's length times the length of b\mathbf{b}.

Check it with the numbers in the viewport. Both arrows have length 10≈3.16\sqrt{10} \approx 3.16, the angle between them is θ≈53.1\tok{theta}{\theta} \approx 53.1°, and cos⁡θ=0.6\cos\theta = 0.6. The shadow is 3.16×0.6≈1.903.16 \times 0.6 \approx 1.90 long, and 1.90×3.16≈61.90 \times 3.16 \approx 6: the same 6 as multiply-then-add.

a⋅b=∣a∣ ∣b∣ cos⁡θ=10⋅10⋅0.6=6\tok{dot}{\mathbf{a}\cdot\mathbf{b}} = \tok{length}{|\mathbf{a}|}\,|\mathbf{b}|\,\tok{cos_theta}{\cos\theta} = \sqrt{10}\cdot\sqrt{10}\cdot 0.6 = 6
Shadow length |a| cos θ, times the length of b.
Why the two recipes always agree1/4
  1. 1
    ∣a−b∣2=(a1−b1)2+(a2−b2)2|\mathbf{a}-\mathbf{b}|^2 = (a_1-b_1)^2 + (a_2-b_2)^2

    Pythagoras for the arrow from the tip of b to the tip of a. Here (3 − 1)² + (1 − 3)² = 8.

Turn b\mathbf{b} to (−1, 3)(-1,\ 3).

Now point b\mathbf{b} away: b=(−1, −2)\mathbf{b} = (-1,\ -2).

Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 9, Dot products and duality

07Cosine similarity, word maps and 3D

To compare directions only, shrink both arrows to length 1 first. The viewport does it: the arrows shrink onto the cyan circle of radius 1 and grow back. An arrow of length 1 is a unit vector, and you get it by dividing a vector by its own length: a^=a/∣a∣\hat{\mathbf{a}} = \mathbf{a} / |\mathbf{a}|.

Here a^=(3, 1)/3.16≈(0.95, 0.32)\hat{\mathbf{a}} = (3,\ 1)/3.16 \approx (0.95,\ 0.32) and b^≈(0.32, 0.95)\hat{\mathbf{b}} \approx (0.32,\ 0.95). Their dot product is 0.95×0.32+0.32×0.95≈0.60.95\times0.32 + 0.32\times0.95 \approx 0.6, which is cos⁡θ\cos\theta. This number is the cosine similarity: 1 for the same direction, 0 for perpendicular, −1-1 for opposite, whatever the lengths.

cos⁡θ=a^⋅b^=a⋅b∣a∣ ∣b∣=610⋅10=0.6\tok{cos_theta}{\cos\theta} = \hat{\mathbf{a}}\cdot\hat{\mathbf{b}} = \frac{\tok{dot}{\mathbf{a}\cdot\mathbf{b}}}{\tok{length}{|\mathbf{a}|}\,|\mathbf{b}|} = \frac{6}{\sqrt{10}\cdot\sqrt{10}} = 0.6
Loading figure
Toy vectors placed by hand: royal words point one way, food words another. Click a word to choose it, click a row to measure its angle.

Word vectors (embeddings) work this way. In the map, king and queen point almost the same way, cosine 0.989, while king and apple are close to perpendicular, 0.079. Switch the ranking to dot product and prince drops from first (0.998) to fourth: it is a short arrow, and the raw dot product rewards length. That is why search compares embeddings by cosine.

Give each vector a third number: a=(2, 1, 2)\mathbf{a} = (2,\ 1,\ 2) and b=(1, 2, 2)\mathbf{b} = (1,\ 2,\ 2).

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 3.1 to 3.4

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Linear algebra: vectors and spaces

Compiling scene
Loading editor

Questions learners ask

What is a vector in machine learning?
A vector is an ordered list of numbers, such as (3, 1) or the hundreds of numbers that describe a word or an image. You can picture it as an arrow from the origin, with one step along each axis per number. Data points, model weights and embeddings are all stored as vectors, so the same few operations (adding, scaling, length and the dot product) appear everywhere in machine learning.
What does the dot product actually tell you?
It measures how much two vectors point the same way, scaled by their lengths. You compute it by multiplying matching components and adding: (3, 1) and (1, 3) give 3 + 3 = 6. It equals the length of one vector times the length of the other times the cosine of the angle between them, so it is positive when they agree, zero when they are perpendicular and negative when they point against each other.
What is the difference between cosine similarity and the dot product?
Cosine similarity is the dot product divided by both lengths, so it depends only on the angle and always lies between -1 and 1. The plain dot product also grows with length, so a long vector can outscore a better-aligned short one. If every vector is first scaled to length 1, the two give the same number, which is why many vector databases normalise embeddings and then use the faster dot product.

Next in Maths Primer

0.3Matrices: tables of numbers that move spaceWatch a grid stretch, rotate and shear as you edit a 2×2 matrix, then see that matrix-vector multiplication is that movement applied to one arrow.

Sources

  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy