4.3 Multi-head attention and the transformer block
A transformer block is a residual stream that every sublayer reads from and adds to: attention moves information between tokens through several narrow heads at once, the MLP rewrites each token on its own, and twelve d × d matrices per layer hold almost all of the parameters.
01Why several heads
Lesson 4.1 built one attention head: for each query, one softmax decides how much of every other token to average in. One softmax gives one weighting per token. The word picked needs two at the same time: the token just before it, to find its subject, and the object that arrives later, block.
Multi-head attention runs heads in parallel. Head projects the -dimensional token vectors into its own -dimensional subspace and computes its own attention pattern there:
The viewport shows the toy block: lesson 4.1's seven tokens with and , so and every head is exactly one of lesson 4.1's heads. On the left is the query matrix with its columns tinted by head. It folds into a stack of four slabs, each slab produces its own attention map on the right, and the head outputs are merged and mixed by .
Every in the cockpit uses the same fused matrices ; only the grouping of columns into softmaxes changes. More heads buy more independent patterns, each computed in a narrower space. Push it to and each head has . Four of the twelve heads are then within bits of uniform ( bits), but not for lack of room: head 2 is exactly uniform because its column of is all zeros (the unused third axis of lesson 4.1's head 0), and heads 9 to 11 are the columns of 4.1's broad head 3, which was already near-uniform at . Heads 0 and 1 stay sharp, with mean row entropies of 1.04 and 1.29 bits. What really costs is direction. A score is a product of two numbers, so every query with ranks the keys in the same order and favours the same largest key, and every query with favours the smallest. In head 8, robot and block have equal keys, so picked splits its weight 0.50 / 0.50 between them.
One big head: .
Twelve one-column heads: , .
Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2.2
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 3.6
02Split and merge heads
Implementations do not loop over heads. One fused matrix computes every head's queries at once, and two reshapes move a head axis in and out. First at the toy's size, the one in the viewport (batch 1, , , ): is , the reshape reads each token's 12 numbers as 4 groups of 3, , and the transpose swaps the token and head axes, : four slabs, one per head. The same moves with GPT-2 small's sizes (, , ):
One (d × d) matmul produces all heads' queries side by side; columns 64k … 64k + 63 belong to head k.
Splits the last axis only. No data moves; each token's 768 numbers are read as 12 groups of 64.
Heads become a batch axis, so Q Kᵀ and the softmax run independently per (b, k). Scores are (B, h, T, T).
After the weighted sums , merge_heads reverses both moves: transpose back to and reshape to . Token 's vector is then the concatenation of its head outputs, and mixes them so every output feature can draw on every head. Hover to see its grid over the merged tensor.
- 1
The per-head projections are column blocks of one d × d matrix, so Q, K and V cost d² weights each for any h.
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Ch. 11.5
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 3.6.2
03The residual stream
A transformer block never replaces its input. Each sublayer reads the current vector of every token, computes something, and adds the result back. The running sum is the residual stream : one -dimensional lane per token, the same shape from the embeddings to the output.
Every sublayer reads and writes this tensor. Its shape never changes between blocks.
The viewport draws the seven tokens as lanes rising through blocks. In each attention sublayer the stream is tapped (the cyan LayerNorm discs sit on the taps), the heads move information sideways between lanes, one beam per query and head from that head's strongest key, coloured as in the heads view, and the result lands on the + junctions. Then the MLP widens each lane on its own, , and adds back. During the forward pass the emerald pulse on each lane climbs straight up while the branch computes: the identity copy costs nothing.
What is in a lane. Next to picked the scene draws its 12 numbers as a row of cells (emerald positive, rose negative) at every junction. As each sublayer finishes, its write appears in the branch, slides onto the lane and the row above it shows the sum: , cell by cell. The attention writes are the toy's real head outputs ( is the identity here, as in the challenge); the MLP has no trained weights in this toy, so its writes come from a fixed random MLP and are illustrative. Pick another token with Probe token in the cockpit; the telemetry line probe ‖x‖ gives the vector's length before and after the stack.
- 1
Every sublayer F (attention or MLP, with its LayerNorm) adds to the stream.
Where position enters, and where the stream ends. Positions are not a separate sublayer. With learned or sinusoidal encodings (lesson 4.2) they are added to at the bottom of the stream; with RoPE they are applied inside every attention sublayer, to the queries and keys only. At the top, the final hidden state of each token is multiplied by the (often tied) embedding matrix to give one logit per vocabulary entry, the step lesson 4.0 follows from logits to the next word.
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 4.4 and 4.5
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Ch. 11.7
The rest of this lesson is in Pro
- 04Pre-norm, post-norm and the MLP
- 05Counting parameters and FLOPs
- 06KV caches, MQA, GQA and causal decoders
- Python challenge: Multi-head attention and a parameter counter
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- Why does multi-head attention split d_model across heads instead of giving each head the full width?
- Splitting keeps the cost fixed: h heads of width d/h use the same d by d projection matrices and the same number of score multiplications as one head of width d. What changes is that the model gets h independent softmaxes, so different heads can attend to different tokens for the same query, for example the previous token and the object of a verb. Very narrow heads lose discriminating power, which is why typical head widths are 64 or 128.
- What is the difference between pre-norm and post-norm transformers?
- Post-norm applies LayerNorm after the residual addition, x plus F(x), as in the original 2017 transformer. Pre-norm applies it to the input of each sublayer, x plus F(LayerNorm(x)), and adds one final LayerNorm before the output layer. Pre-norm keeps the residual path an identity from the loss to the embeddings, so deep stacks train stably with little warm-up; nearly all large language models use it or its RMSNorm variant.
- How many parameters does a transformer layer have?
- About 12 times d squared, where d is the model width: 4 d squared for the query, key, value and output projections and 8 d squared for an MLP that widens to 4d. Biases and LayerNorms add 13d, which is negligible. A whole decoder has roughly 12 L d squared plus V d parameters for L layers and a vocabulary of V tokens; GPT-2 small, with d = 768, L = 12 and V = 50,257, comes to about 124 million.
Next in Transformers and LLMs
4.4Pretraining, sampling and fine-tuning LLMsNext-token loss at scale, temperature and nucleus sampling, and parameter-efficient fine-tuning with LoRA.Sources
- Attention Is All You Need, Vaswani et al., 2017
- Build a Large Language Model (From Scratch), Sebastian Raschka
- Maths, CS & AI Compendium, Henry Ndubuaku
- Dive into Deep Learning, Zhang, Lipton, Li & Smola