Skip to content
LearnMLByDoing
Free

4.0 Inside an LLM: from one neuron to the next word

45 minFree lessonintermediateGPT-2CircuitsInduction headsAlammarTransformer ExplainerVaswaniKarpathyDLNucleusRaschka

Follow a real 27,744-parameter GPT from one neuron to an attention head, a transformer block and the loop that writes text, with every number computed live in your browser.

01One neuron: a weighted vote, then a bend

A neuron multiplies each input by a weight, adds the products and a bias, and bends the sum with an activation function. The neuron in the viewport is not a toy. It is neuron 19 in layer 1 of a real two-block GPT running in your browser, reading the 32 numbers that currently describe the word lost in the little girl also lost.

z=w⋅x+b=∑i=132wixi+b,h=GELU⁡(z)\tok{neuron}{z = \mathbf{w}\cdot\mathbf{x} + b} = \sum_{i=1}^{32} w_i x_i + b, \qquad h = \tok{act}{\operatorname{GELU}}(z)
Each edge into the sphere is one weight wiw_i (emerald positive, rose negative, thicker when larger); the pulses travelling along the edges are the products wixiw_i x_i.
Loading figure
The same neuron, flat. Hover a bar to read its xix_i, wiw_i and product; the sliders edit the real model, so the her and his bars are the model's actual next-word probabilities.

At lost the 32 products add to +1.41+1.41 and the bias is −0.30-0.30, so z=+1.11z = +1.11 and GELU⁡(z)=+0.96\operatorname{GELU}(z) = +0.96. The largest single vote is dimension 13: x13=−3.19x_{13} = -3.19 times w13=−0.22w_{13} = -0.22 gives +0.70+0.70. On the old king never found the same neuron gets z=−0.05z = -0.05 and outputs −0.02-0.02: it is off. GELU behaves like ReLU far from zero, but it is smooth and dips slightly below zero for small negative inputs.

Make the king find her hat: switch to the old king never found and raise neuron 19's bias by 4.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.3, hidden units

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd: a neuron

02Many neurons: the MLP works on one position at a time

Layer 1 has 128 neurons like the last one, all reading the same 32 inputs. At lost, 38 of them output more than 0.1. Their 128 outputs are mixed back down to 32 numbers by a second weight matrix, and that vector is added to the token's running vector.

MLP(x)=GELU⁡(x Wfc+bfc) Wproj+bproj\mathrm{MLP}(\mathbf{x}) = \tok{act}{\operatorname{GELU}}(\mathbf{x}\,W_{fc} + \mathbf{b}_{fc})\,W_{proj} + \mathbf{b}_{proj}
Column 19 of WfcW_{fc} is the weight vector w\mathbf{w} of the previous step; row 19 of WprojW_{proj} is what the gain slider scales.
W_fc[,]4,096 values

Expand: one column per hidden neuron.

W_proj[,]4,096 values

Contract: one row per neuron, its vote on each of the 32 stream dimensions. With biases the MLP holds 8,352 of the block's 12,704 parameters.

Click any dot in the grid to make it the editable neuron of the previous step; the bias and gain sliders then act on that neuron instead. Its size and colour are its GELU output at the focused position.

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.3, position-wise feed-forward networks

Language Models are Unsupervised Multitask Learners, Radford et al., 2019, Sec. 2.3, model

03Attention: queries, keys and values with real numbers

Before any block runs, each token becomes a vector: its row of the embedding table plus a row for its position, xt=WE[idt]+WP[t]\tok{embed}{\mathbf{x}_t = W_E[\text{id}_t] + W_P[t]}. Every head then projects each normalised vector three ways: a query (what am I looking for), a key (what can I be found by) and a value (what I pass on if chosen). Lesson 4.1 builds the geometry of these vectors and the variance argument for dk\sqrt{d_k}; here we follow one real head, L1H0, on today anna lee met tom ross . tom.

One row of attention, at the final tom1/4
  1. 1
    sij=qi⋅kj=∑c=18qic kjc\tok{score}{s_{ij}} = \tok{query}{\mathbf{q}_i}\cdot\tok{key}{\mathbf{k}_j} = \sum_{c=1}^{8} q_{ic}\,k_{jc}

    Eight products per cell of the score grid. At the final tom (i=8i = 8) the key of ross scores 14.35, lee 5.01, anna −4.74.

So the head at the final tom reads from ross, the word that followed the first tom, and the model goes on to predict ross with p=0.998p = 0.998. Press play to watch the row being built: query and key strips light in turn, each product pulse raises one grid cell, the row shrinks by 8\sqrt 8, the future cells hatch rose, the softmax bars grow and the values flow into oi\mathbf{o}_i.

Q, K, V of one head[,]72 values

Head h owns columns 8h to 8h + 7 of the 32-column W_Q, W_K and W_V. The scores and weights are T × T = 9 × 9, one row per query.

Loading figure
The focused head as an exact table. Hover a cell to light its query and key in the viewport and read q⋅k\mathbf{q}\cdot\mathbf{k}, the scaled score and the weight; click a row label to make that token the query.

Switch the causal mask off and let every token read the whole sentence.

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.2.1, scaled dot-product attention

The Illustrated Transformer, Jay Alammar, Self-attention in detail

04Heads that work together: the induction circuit

In ben moss met tom lee . tom the surname after the second tom cannot be memorised: first names and surnames were paired at random in training. The model has to find where tom appeared before and copy what came after it. It answers lee with p=0.997p = 0.997, using two heads in two different layers.

Step one, layer 0. Head L0H2 at lee puts 0.92 of its attention on tom, the word before it, and writes that into lee's vector: the stream at lee now carries the tag *the word before me was tom*. Step two, layer 1. Head L1H0 at the second tom searches for a key carrying that tag, finds lee (0.91) and copies it forward to the last position. The viewport shows layer 1; set Block to layer 0 in the parameters to see L0H2's map.

Zero L0H2, the head that writes the tags.

Zero L1H0, the head that does the lookup, instead.

In-context Learning and Induction Heads, Olsson et al., 2022, Induction heads and K-composition

A Mathematical Framework for Transformer Circuits, Elhage et al., 2021, Two-layer attention-only transformers

05The transformer block: a residual highway with two on-ramps

Zoom out and the same pattern holds for the whole block. A block never overwrites a token's vector, the residual stream. It reads a normalised copy, computes something, and adds the result back. Attention is the only on-ramp that moves information sideways between lanes; the MLP works up each lane on its own.

x←x+Attn(LN⁡1(x))\tok{resid}{\mathbf{x}} \leftarrow \mathbf{x} + \mathrm{Attn}\big(\tok{ln}{\operatorname{LN}_1}(\mathbf{x})\big)
x←x+MLP(LN⁡2(x))\mathbf{x} \leftarrow \mathbf{x} + \mathrm{MLP}\big(\operatorname{LN}_2(\mathbf{x})\big)
Pre-LN, as in GPT-2: each LayerNorm rescales only the copy handed to its sub-layer, never the stream itself.

At lost the stream's length grows from 1.11 after the embedding to 1.42 after block 0 and 1.91 after block 1: each sub-layer adds a little, and nothing is thrown away. Inside attention, the 4 heads each produce 8 numbers per token. They are concatenated back to 32 and mixed by the output projection WO\tok{w_o}{W_O} before the add.

concat of head outputs[,]192 values

Then × W_O [32 × 32] + b_O. Zeroing a head in the viewport zeroes its 8 columns here.

Zero L1H0 on this prompt. It was the head that carried tom to the end; here it should carry girl to lost.

The chips beside the last lane are the logit lens (toggle Logit lens): the word the model would predict if it stopped at that snapshot. At lost: lost 0.35 after the embedding, her 0.23 after block 0, her 0.75 after layer-1 attention, her 1.00 at the output. Lesson 4.3 takes the residual stream further, with its own lab on how blocks read and write it.

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.1, encoder and decoder stacks

Language Models are Unsupervised Multitask Learners, Radford et al., 2019, Sec. 2.3, pre-activation LayerNorm

06The whole model: from table lookup to next-word probabilities

Stack the two blocks, apply a final LayerNorm, and the last position's vector has to become one score per word. This model reuses the table that embedded the words: a word's score is the dot product of the final vector with that word's row of WEW_E, the same similarity test as q⋅k\mathbf{q}\cdot\mathbf{k}. GPT-2 ties its embeddings the same way.

z=LN⁡f(xT) WE⊤,p(w∣context)=softmax⁡(z/τ)w\tok{logits}{\mathbf{z} = \operatorname{LN}_f(\mathbf{x}_T)\,W_E^{\top}}, \qquad \tok{prob}{p(w \mid \text{context})} = \tok{softmax}{\operatorname{softmax}}(\mathbf{z}/\tok{temp}{\tau})_w
The wall at the back is WEW_E, one column per word. Each column lights by its logit and the probability bars rise from the top of the wall.
W_E (embedding and unembedding)[,]1,888 values

27,744 parameters in total: W_E 1,888 + W_P 384 + 2 × 12,704 per block + 64 for the final LayerNorm. GPT-2 small has about 124 million, built the same way with 12 blocks of 12 heads and d_model = 768.

Read the lens. If the model stopped after block 0 it would say lee (0.54), the wrong surname: by then it knows a surname is due, not which one. Layer-1 attention fixes it to ross (0.79), and the layer-1 MLP finishes the job (0.998).

Loading figure
Each cell applies the final LayerNorm and WEW_E to one residual snapshot and shows the top word. Change the prompt or zero a head and the table updates.

Temperature divides every logit before the softmax. For the first word of a text (prompt <bos>) the logits are the 7.48, today 6.93, lily 5.58. At τ=1\tau = 1 that gives the 0.413; at τ=0.3\tau = 0.3, 0.855; at τ=3\tau = 3, 0.158. The ranking never changes, only how sharply probability concentrates on the top.

Loading figure
The real first-word logits. τ\tau and top-k here are the viewport's sliders, so the 3D probability bars move with them; words cut by top-k turn rose.

Language Models are Unsupervised Multitask Learners, Radford et al., 2019, Sec. 2.3, model

The Curious Case of Neural Text Degeneration, Holtzman et al., 2020

07Generation: the loop that writes the text

Generation is the whole model in a loop: compute the next-word distribution, draw a word, append it, run again. The run stops when the model predicts <bos>, which it learned as its end-of-text marker, or when the 12-token context is full.

The dart is a uniform draw uu from a seeded generator; the chosen word is the bar it lands in on the cumulative strip. With seed 3 at τ=1\tau = 1 the model writes mia gave the key to lily . lily thanked mia . Greedy decoding (τ=0\tau = 0) always takes the tallest bar and always writes the little bird ran to the park ., starting with the because it has the largest first-word logit.

Overheat the sampler: τ=2.5\tau = 2.5 with seed 5.

Loading figure
Write your own prompt from the model's 59 words; the viewport switches to it. Click a suggestion here, or a probability bar in the viewport, to choose the next word yourself.

Decoding strategies beyond temperature and top-k, such as top-p, are covered in lesson 4.4.

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 4.7 and 5.3, generating text

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, Let's build GPT

Compiling scene
Loading editor

Questions learners ask

How does an LLM predict the next word?
It turns each token into a vector, lets every position gather information from earlier positions with attention, refines each vector with an MLP, and repeats this for every block. The final vector at the last position is compared with the embedding of every word in the vocabulary; those similarity scores, the logits, go through a softmax to give a probability for each possible next word. Generation draws one word, appends it and runs the model again.
What is an induction head?
A two-step attention circuit that continues a pattern seen earlier in the prompt. A head in an early layer copies information about the previous token into each position. A head in a later layer then searches for the position whose previous token matches the current token and copies the word found there. In this lesson's model, L0H2 and L1H0 form one: given a repeated first name, they retrieve the surname that followed it the first time.
What does temperature do in an LLM?
It divides the logits before the softmax. A temperature below 1 sharpens the distribution toward the most likely word, a temperature above 1 flattens it so unlikely words are sampled more often, and a temperature near 0 always picks the most likely word. It never changes the model's weights or the ranking of the words, only how the next word is drawn.

Next in Transformers and LLMs

4.1Self-attention geometry and scaled dot-product in 3DProject tokens into query, key and value spaces, fire attention beams between them, and see exactly why we divide by √dₖ.

Sources

  • Language Models are Unsupervised Multitask Learners, Radford et al., 2019
  • A Mathematical Framework for Transformer Circuits, Elhage et al., 2021
  • In-context Learning and Induction Heads, Olsson et al., 2022
  • The Illustrated Transformer, Jay Alammar
  • Transformer Explainer, Cho et al., 2024, Georgia Tech Polo Club
  • Attention Is All You Need, Vaswani et al., 2017
  • Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy
  • Deep Learning, Goodfellow, Bengio & Courville
  • The Curious Case of Neural Text Degeneration, Holtzman et al., 2020
  • Build a Large Language Model (From Scratch), Sebastian Raschka