Skip to content
LearnMLByDoing
Free

4.1 Self-attention geometry and scaled dot-product in 3D

42 minPro, with free previewintermediateVaswaniRaschka

Self-attention lets every token ask a question of every other token, and a dot product in a learned space decides who answers.

01Queries, keys and values: three views of every token

A transformer layer receives a sequence of NN token embeddings stacked as the rows of X∈RN×dmodelX \in \mathbb{R}^{N \times d_{\text{model}}}. Self-attention rebuilds each row by pulling in information from the other rows. The whole mechanism comes down to one question: for token ii, which other tokens matter, and by how much?

Attention answers it by projecting every token three times with learned matrices. The query qi\tok{query}{\mathbf{q}_i} encodes what token ii is looking for, the key kj\tok{key}{\mathbf{k}_j} encodes what token jj can be found by, and the value vj\tok{value}{\mathbf{v}_j} is what token jj hands over once it is selected.

Q=XWQ,K=XWK,V=XWV,WQ,WK∈Rdmodel×dk\tok{query}{Q} = X W_Q, \qquad \tok{key}{K} = X W_K, \qquad \tok{value}{V} = X W_V, \qquad W_Q, W_K \in \mathbb{R}^{d_{\text{model}} \times \tok{head_dim}{d_k}}
Three linear maps of the same embeddings. Only the matrices differ.
X[,]35 values

the robot picked up the red block: 2 positional features plus noun, modifier and verb flags

Q, K[,]21 values

a 3-dimensional head, so every query and key in the scene is a real arrow

The score of key jj for query ii is their dot product, qi⋅kj=∥qi∥ ∥kj∥cos⁡θij\tok{query}{\mathbf{q}_i} \cdot \tok{key}{\mathbf{k}_j} = \lVert \mathbf{q}_i \rVert \, \lVert \mathbf{k}_j \rVert \cos\theta_{ij} (lesson 0.2, Vectors, builds this identity). In the scene the amber arrow is the query of the selected token (picked) and the thin cyan arrows are the keys of all seven tokens. A key that points the same way as the query, and is long, scores high.

A worked example small enough to do by hand. Take three tokens, the, cat and sat, in a 2-dimensional head. The query of sat is q=(1,2)\mathbf{q} = (1, 2) and the keys are kthe=(1,0)\mathbf{k}_{\text{the}} = (1, 0), kcat=(0,2)\mathbf{k}_{\text{cat}} = (0, 2), ksat=(1,1)\mathbf{k}_{\text{sat}} = (1, 1). The dot products are 1⋅1+2⋅0=11 \cdot 1 + 2 \cdot 0 = 1, 1⋅0+2⋅2=41 \cdot 0 + 2 \cdot 2 = 4 and 1⋅1+2⋅1=31 \cdot 1 + 2 \cdot 1 = 3. Dividing by dk=2\sqrt{d_k} = \sqrt{2} gives scores 0.71, 2.83 and 2.12. Exponentiate: 2.03, 16.92 and 8.34, which add up to 27.29. Divide each by that total and the weights are 0.07, 0.62 and 0.31: they are positive and sum to 1, and cat, the key that points most nearly along the query, gets the most.

Loading figure
The worked example, live. Drag the amber query: each key's score sits at its tip, the bars are the softmax, and the table repeats the arithmetic above. Turn on the values: the emerald z=∑jajvj\mathbf{z} = \sum_j a_j \mathbf{v}_j starts at (0.82, 0.07) and, wherever you drag q\mathbf{q}, never leaves the shaded triangle spanned by the three values, because the weights are positive and sum to 1.

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 3.3–3.4

02From dot products to attention beams

One matrix product computes every score at once: QK⊤QK^{\top} is an N×NN \times N table whose row ii says how well each key matches query ii. A softmax across that row turns scores into weights, and the weights mix the values.

One row of attention1/4
  1. 1
    sij=qi⋅kjdks_{ij} = \frac{\tok{query}{\mathbf{q}_i} \cdot \tok{key}{\mathbf{k}_j}}{\sqrt{\tok{head_dim}{d_k}}}

    One score per (query, key) pair. The division keeps scores on a unit scale; step 3 explains why.

Reading the scene. While the animation plays, the scene walks through one row: each key lights up in turn with its score ss at its tip, then the softmax turns the scores into beams, then value pulses flow in and the emerald zi\mathbf{z}_i grows (the telemetry line step says which stage is showing; the cockpit's *Animate the computation* switches to the finished row only). Each beam runs from the query to key jj; its cross-section is proportional to aija_{ij} and the heaviest beam is emerald. A ring around the query shows how much it attends to itself. Pulses travel from the keys into the query because that is the direction values flow. The emerald arrow is zi\tok{context}{\mathbf{z}_i}, a weighted average of the value vectors, so it always lies inside their convex hull. Hover zi\tok{context}{\mathbf{z}_i} to see this: the scene redraws every value arrow from the query, and the tip of zi\mathbf{z}_i lands among their tips, pulled toward the heaviest one.

Loading figure
Row i=5i = 5 (red) of head 1: the scaled scores s5js_{5j} under each key and the weights a5ja_{5j} above. The slider is the scene's temperature τ\tau, so the beams follow it. At τ=1\tau = 1, block takes 0.89 of the weight.

The bar strip shows one row. Stack all seven rows and you have the attention matrix A=softmax⁡(QK⊤/dk)A = \operatorname{softmax}(QK^{\top}/\sqrt{d_k}), the picture most papers draw. The matrix below is computed from the scene's head, temperature, scaling and mask, so it changes with the cockpit; click a row label to make that token the query in the scene.

Loading figure
Rows are queries, columns are keys, and every row sums to 1. The amber row is the scene's query and its outlined cell is the heaviest beam. Switch to scores s to see the numbers before the softmax.

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2.1, Eq. 1

The rest of this lesson is in Pro

  • 03Why divide by √dₖ: a variance argument
  • 04Causal masking: no peeking at the future
  • 05Multi-head attention: several geometries at once
  • Python challenge: Scaled dot-product attention from scratch
  • Knowledge check, 3 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is self-attention in a transformer?
Self-attention computes, for every token, a weighted average of value vectors from all tokens in the sequence. The weights come from a softmax over dot products between that token's query vector and every token's key vector, where queries, keys and values are learned linear projections of the embeddings.
Why is scaled dot-product attention divided by the square root of d_k?
If query and key components have unit variance, their dot product has variance d_k. Dividing by √d_k restores unit variance, which keeps the softmax away from saturation, where its gradients vanish and training stalls.
What does a causal mask do in attention?
A causal mask adds −∞ to the scores of all future positions (key index greater than query index) before the softmax. Those weights become exactly zero and the rest renormalise, so a decoder can be trained on every position in parallel without seeing the tokens it must predict.

Next in Transformers and LLMs

4.2Tokenisation and embedding spacesByte-pair encoding from scratch, and how embedding tables turn token ids into directions with meaning.

Sources

  • Attention Is All You Need, Vaswani et al., 2017
  • Build a Large Language Model (From Scratch), Sebastian Raschka