0.2 Vectors: arrows, adding, scaling and the dot product
A vector is an arrow and a list of numbers at the same time, and the dot product turns two of them into one number that says how much they point the same way.
01A vector is an arrow and a list
Watch the cyan dot in the viewport: it walks 3 squares right, then 1 square up, and then the emerald arrow shoots from the centre (the origin) straight to where the dot stopped. That arrow is a vector: something with a direction and a length. Where it starts does not matter; slide it anywhere and it is the same vector.
The two moves are all you need to rebuild the arrow, so we can also write the vector as the list of moves: . The first number is the step along (right is positive, left is negative), the second the step along (up is positive). The label at the tip shows both views at once.
Give the first number a minus sign: .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 1, Vectors
02Length: Pythagoras on the two moves
The two moves and the arrow make a right-angled triangle: 3 along, 4 up, and the arrow is the long side. How long is it? Hover to draw a circle of that radius around the origin: every arrow that ends on the circle has the same length.
Pythagoras says the long side, squared, equals the two short sides squared and added: . The length is the number that gives 25 when multiplied by itself, , and the tip label reads .
Go back to .
Flip both signs: .
03Adding: one trip, then the other
Adding two vectors means doing one trip and then the other. In the viewport the amber arrow slides until its tail sits on the tip of , and the cyan arrow from the origin to where you end up is the sum . This is called adding tip to tail.
Because each vector is a list of moves, you add the moves: right , up , so . Its length is , a little less than , because the route bends.
After the sum appears, a faint emerald line traces the other order: first, then . It lands on the same point, so the order of adding does not matter, and the two routes outline a parallelogram.
Make , the exact reverse of .
04Scaling: stretch, shrink and flip
Multiplying a vector by an ordinary number stretches it along its own line. In the viewport the cyan arrow grows from to and back, while the faint emerald arrow keeps the original for comparison.
With and , every component doubles: . The direction stays the same and the length doubles, from to . A between 0 and 1 shrinks instead: .
Set .
Subtraction is adding a flipped vector: . For that is , the arrow from the tip of to the tip of .
05The dot product: multiply, then add
The dot product turns two arrows into one number that measures how much they point the same way. The arc between them in the viewport is coloured by it: emerald when the arrows agree, grey when they are at a right angle, rose when they point against each other.
The recipe uses the lists: multiply matching components, then add the results. For and that is . The widget lights up each product in turn, and its buttons move the arrows in the viewport.
Let turn around the origin while stays put.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 3.2, Inner products
06Angle and shadow: why it measures agreement
Shine a light straight down onto the line through (the faint amber line): casts a shadow, the grey bar, and the dashed drop meets the line at a right angle. The more leans toward , the longer the shadow. The dot product is the shadow's length times the length of .
Check it with the numbers in the viewport. Both arrows have length , the angle between them is °, and . The shadow is long, and : the same 6 as multiply-then-add.
- 1
Pythagoras for the arrow from the tip of b to the tip of a. Here (3 − 1)² + (1 − 3)² = 8.
Turn to .
Now point away: .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 9, Dot products and duality
07Cosine similarity, word maps and 3D
To compare directions only, shrink both arrows to length 1 first. The viewport does it: the arrows shrink onto the cyan circle of radius 1 and grow back. An arrow of length 1 is a unit vector, and you get it by dividing a vector by its own length: .
Here and . Their dot product is , which is . This number is the cosine similarity: 1 for the same direction, 0 for perpendicular, for opposite, whatever the lengths.
Word vectors (embeddings) work this way. In the map, king and queen point almost the same way, cosine 0.989, while king and apple are close to perpendicular, 0.079. Switch the ranking to dot product and prince drops from first (0.998) to fourth: it is a short arrow, and the raw dot product rewards length. That is why search compares embeddings by cosine.
Give each vector a third number: and .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 3.1 to 3.4
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Linear algebra: vectors and spaces
Questions learners ask
- What is a vector in machine learning?
- A vector is an ordered list of numbers, such as (3, 1) or the hundreds of numbers that describe a word or an image. You can picture it as an arrow from the origin, with one step along each axis per number. Data points, model weights and embeddings are all stored as vectors, so the same few operations (adding, scaling, length and the dot product) appear everywhere in machine learning.
- What does the dot product actually tell you?
- It measures how much two vectors point the same way, scaled by their lengths. You compute it by multiplying matching components and adding: (3, 1) and (1, 3) give 3 + 3 = 6. It equals the length of one vector times the length of the other times the cosine of the angle between them, so it is positive when they agree, zero when they are perpendicular and negative when they point against each other.
- What is the difference between cosine similarity and the dot product?
- Cosine similarity is the dot product divided by both lengths, so it depends only on the angle and always lies between -1 and 1. The plain dot product also grows with length, so a long vector can outscore a better-aligned short one. If every vector is first scaled to length 1, the two give the same number, which is why many vector databases normalise embeddings and then use the faster dot product.
Sources
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Khan Academy: Algebra, Calculus and Statistics, Khan Academy