Skip to content
LearnMLByDoing
Free

4.3 Multi-head attention and the transformer block

38 minProadvancedVaswaniRaschkaCompendiumD2L

A transformer block is a residual stream that every sublayer reads from and adds to: attention moves information between tokens through several narrow heads at once, the MLP rewrites each token on its own, and twelve d × d matrices per layer hold almost all of the parameters.

01Why several heads

Lesson 4.1 built one attention head: for each query, one softmax decides how much of every other token to average in. One softmax gives one weighting per token. The word picked needs two at the same time: the token just before it, to find its subject, and the object that arrives later, block.

Multi-head attention runs h\tok{heads}{h} heads in parallel. Head kk projects the d\tok{d_model}{d}-dimensional token vectors into its own dk=d/h\tok{head_dim}{d_k} = d/h-dimensional subspace and computes its own attention pattern there:

head⁡k=softmax⁡ ⁣(QkKk⊤dk)Vk,Qk=XWQ(k), Kk=XWK(k), Vk=XWV(k)∈RT×dk\operatorname{head}_k = \operatorname{softmax}\!\left(\frac{Q_k K_k^{\top}}{\sqrt{\tok{head_dim}{d_k}}}\right) V_k, \qquad Q_k = X W_Q^{(k)},\ K_k = X W_K^{(k)},\ V_k = X W_V^{(k)} \in \mathbb{R}^{T \times \tok{head_dim}{d_k}}
Each head has its own projections and its own softmax; k = 0 … h − 1.

The viewport shows the toy block: lesson 4.1's seven tokens with d=12d = 12 and h=4h = 4, so dk=3d_k = 3 and every head is exactly one of lesson 4.1's heads. On the left is the 7×127 \times 12 query matrix with its columns tinted by head. It folds into a stack of four 7×37 \times 3 slabs, each slab produces its own 7×77 \times 7 attention map on the right, and the head outputs are merged and mixed by WO\tok{w_o}{W_O}.

Every hh in the cockpit uses the same fused matrices WQ,WK,WV∈R5×12W_Q, W_K, W_V \in \mathbb{R}^{5 \times 12}; only the grouping of columns into softmaxes changes. More heads buy more independent patterns, each computed in a narrower space. Push it to h=12h = 12 and each head has dk=1d_k = 1. Four of the twelve heads are then within 0.060.06 bits of uniform (log⁡27=2.81\log_2 7 = 2.81 bits), but not for lack of room: head 2 is exactly uniform because its column of WQW_Q is all zeros (the unused third axis of lesson 4.1's head 0), and heads 9 to 11 are the columns of 4.1's broad head 3, which was already near-uniform at h=4h = 4. Heads 0 and 1 stay sharp, with mean row entropies of 1.04 and 1.29 bits. What dk=1d_k = 1 really costs is direction. A score qikjq_i k_j is a product of two numbers, so every query with qi>0q_i > 0 ranks the keys in the same order and favours the same largest key, and every query with qi<0q_i < 0 favours the smallest. In head 8, robot and block have equal keys, so picked splits its weight 0.50 / 0.50 between them.

One big head: h=1h = 1.

Twelve one-column heads: h=12h = 12, dk=1d_k = 1.

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2.2

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 3.6

02Split and merge heads

Implementations do not loop over heads. One fused matrix computes every head's queries at once, and two reshapes move a head axis in and out. First at the toy's size, the one in the viewport (batch 1, T=7T = 7, d=12d = 12, h=4h = 4): QQ is 1×7×121 \times 7 \times 12, the reshape reads each token's 12 numbers as 4 groups of 3, 1×7×4×31 \times 7 \times 4 \times 3, and the transpose swaps the token and head axes, 1×4×7×31 \times 4 \times 7 \times 3: four 7×37 \times 3 slabs, one per head. The same moves with GPT-2 small's sizes (d=768d = 768, h=12h = 12, dk=64d_k = 64):

Q = X W_Q[,,]6,291,456 values

One (d × d) matmul produces all heads' queries side by side; columns 64k … 64k + 63 belong to head k.

reshape[,,,]6,291,456 values

Splits the last axis only. No data moves; each token's 768 numbers are read as 12 groups of 64.

transpose(0, 2, 1, 3)[,,,]6,291,456 values

Heads become a batch axis, so Q Kᵀ and the softmax run independently per (b, k). Scores are (B, h, T, T).

After the weighted sums AkVkA_k V_k, merge_heads reverses both moves: transpose back to (B,T,h,dk)(B, T, h, d_k) and reshape to (B,T,d)(B, T, d). Token tt's vector is then the concatenation of its hh head outputs, and WO∈Rd×d\tok{w_o}{W_O} \in \mathbb{R}^{d \times d} mixes them so every output feature can draw on every head. Hover WO\tok{w_o}{W_O} to see its grid over the merged tensor.

MultiHead⁡(X)=[head⁡0; head⁡1; … ; head⁡h−1] WO\operatorname{MultiHead}(X) = \big[\operatorname{head}_0;\ \operatorname{head}_1;\ \dots;\ \operatorname{head}_{h-1}\big]\,\tok{w_o}{W_O}
Concatenate along the feature axis, then one d × d projection.
h heads of width d/h cost the same as one head of width d1/4
  1. 1
    WQ=[WQ(0)⋯WQ(h−1)],h⋅d⋅dh=d2W_Q = \big[W_Q^{(0)} \cdots W_Q^{(h-1)}\big],\quad h \cdot d \cdot \tfrac{d}{h} = d^2

    The per-head projections are column blocks of one d × d matrix, so Q, K and V cost d² weights each for any h.

Loading figure
The toy's QQ: rows are tokens, the number in each cell is its feature index, the colour its token. With reshape + transpose, head 0 (outlined) holds features 0 to 2 of all seven tokens. Switch to reshape only: head 0 now holds every feature of the and most of robot, and its rows are no longer tokens. The heads selector is the scene's hh.

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Ch. 11.5

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 3.6.2

03The residual stream

A transformer block never replaces its input. Each sublayer reads the current vector of every token, computes something, and adds the result back. The running sum is the residual stream xℓ\tok{residual}{x_\ell}: one dd-dimensional lane per token, the same shape from the embeddings to the output.

x′=xℓ+MultiHead⁡(LN⁡1(xℓ)),xℓ+1=x′+MLP⁡(LN⁡2(x′))x' = \tok{residual}{x_\ell} + \operatorname{MultiHead}\big(\tok{layernorm}{\operatorname{LN}}_1(\tok{residual}{x_\ell})\big), \qquad \tok{residual}{x_{\ell+1}} = x' + \tok{mlp}{\operatorname{MLP}}\big(\tok{layernorm}{\operatorname{LN}}_2(x')\big)
One pre-norm block: two sublayers, each read → transform → add.
x_ℓ (residual stream)[,,]

Every sublayer reads and writes this tensor. Its shape never changes between blocks.

The viewport draws the seven tokens as lanes rising through L=2L = 2 blocks. In each attention sublayer the stream is tapped (the cyan LayerNorm discs sit on the taps), the heads move information sideways between lanes, one beam per query and head from that head's strongest key, coloured as in the heads view, and the result lands on the + junctions. Then the MLP widens each lane on its own, d→4d→dd \to 4d \to d, and adds back. During the forward pass the emerald pulse on each lane climbs straight up while the branch computes: the identity copy costs nothing.

What is in a lane. Next to picked the scene draws its 12 numbers as a row of cells (emerald positive, rose negative) at every junction. As each sublayer finishes, its write appears in the branch, slides onto the lane and the row above it shows the sum: x′=x+F(x)x' = x + F(x), cell by cell. The attention writes are the toy's real head outputs (WOW_O is the identity here, as in the challenge); the MLP has no trained weights in this toy, so its writes come from a fixed random MLP and are illustrative. Pick another token with Probe token in the cockpit; the telemetry line probe ‖x‖ gives the vector's length before and after the stack.

Unrolling the stream1/3
  1. 1
    xℓ+1=xℓ+Fℓ(xℓ)x_{\ell+1} = x_\ell + F_\ell(x_\ell)

    Every sublayer F (attention or MLP, with its LayerNorm) adds to the stream.

Where position enters, and where the stream ends. Positions are not a separate sublayer. With learned or sinusoidal encodings (lesson 4.2) they are added to x0x_0 at the bottom of the stream; with RoPE they are applied inside every attention sublayer, to the queries and keys only. At the top, the final hidden state of each token is multiplied by the (often tied) embedding matrix to give one logit per vocabulary entry, the step lesson 4.0 follows from logits to the next word.

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 4.4 and 4.5

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Ch. 11.7

The rest of this lesson is in Pro

  • 04Pre-norm, post-norm and the MLP
  • 05Counting parameters and FLOPs
  • 06KV caches, MQA, GQA and causal decoders
  • Python challenge: Multi-head attention and a parameter counter
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

Why does multi-head attention split d_model across heads instead of giving each head the full width?
Splitting keeps the cost fixed: h heads of width d/h use the same d by d projection matrices and the same number of score multiplications as one head of width d. What changes is that the model gets h independent softmaxes, so different heads can attend to different tokens for the same query, for example the previous token and the object of a verb. Very narrow heads lose discriminating power, which is why typical head widths are 64 or 128.
What is the difference between pre-norm and post-norm transformers?
Post-norm applies LayerNorm after the residual addition, x plus F(x), as in the original 2017 transformer. Pre-norm applies it to the input of each sublayer, x plus F(LayerNorm(x)), and adds one final LayerNorm before the output layer. Pre-norm keeps the residual path an identity from the loss to the embeddings, so deep stacks train stably with little warm-up; nearly all large language models use it or its RMSNorm variant.
How many parameters does a transformer layer have?
About 12 times d squared, where d is the model width: 4 d squared for the query, key, value and output projections and 8 d squared for an MLP that widens to 4d. Biases and LayerNorms add 13d, which is negligible. A whole decoder has roughly 12 L d squared plus V d parameters for L layers and a vocabulary of V tokens; GPT-2 small, with d = 768, L = 12 and V = 50,257, comes to about 124 million.

Next in Transformers and LLMs

4.4Pretraining, sampling and fine-tuning LLMsNext-token loss at scale, temperature and nucleus sampling, and parameter-efficient fine-tuning with LoRA.

Sources

  • Attention Is All You Need, Vaswani et al., 2017
  • Build a Large Language Model (From Scratch), Sebastian Raschka
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Dive into Deep Learning, Zhang, Lipton, Li & Smola