Skip to content
LearnMLByDoing
Free

4.4 Pretraining, sampling and fine-tuning LLMs

40 minProintermediateRaschkaNucleusLoRAChinchillaVaswani

A language model is trained to put probability on the next token, decoded by reshaping that distribution with temperature, top-k and top-p, and adapted to new tasks by learning a low-rank correction to its frozen weights.

01Next-token prediction: cross-entropy and perplexity

A GPT-style model is trained on one task: given the tokens so far, give every token in the vocabulary a probability of coming next. The last layer outputs one score per vocabulary entry, the logits z\tok{logits}{z}, and a softmax turns them into a distribution.

Three tokens first. Suppose only block, box and ball were possible, with logits 3.2, 2.7 and 2.5 (the viewport's top three). Exponentiate each: e3.2=24.53e^{3.2} = 24.53, e2.7=14.88e^{2.7} = 14.88, e2.5=12.18e^{2.5} = 12.18. Add them: 51.59. Divide each by the total: 0.475, 0.288 and 0.236. The results are positive and sum to 1, and the biggest logit gets the biggest share. That is the whole softmax; lesson 0.7, Exponentials, logarithms, sigmoid and softmax, builds it up step by step. With the full vocabulary of VV tokens it reads:

pi=ezi∑j=1Vezj,i=1,…,V\tok{prob}{p_i} = \frac{e^{\tok{logits}{z_i}}}{\sum_{j=1}^{V} e^{\tok{logits}{z_j}}}, \qquad i = 1, \dots, V
One distribution per position. The viewport shows 30 candidates for the context in the card at the bottom left.
logits[,,]411,705,344 values

GPT-2's vocabulary has 50,257 tokens and every position predicts a full row. The scene keeps 30 candidates so each one gets a bar.

The viewport is the output after The robot picked up the, sorted by rank. block gets 0.262, box 0.159 and ball 0.130; idioms such as phone (0.072) and pace (0.043) follow. Why draw at random instead of always taking block? A model that always picks the top token writes the same sentence every time, and greedy text tends to fall into loops; sampling keeps the variety the model learned. The amber particles are draws from this distribution, and the thin amber tick beside each bar is how often that token has been drawn so far. Watch the ticks converge to the bar heights.

Loading figure
How a sampler turns pp into one token. Lay the probabilities end to end on a strip from 0 to 1, draw a uniform random number uu, and take the segment it lands in: block owns 0 to 0.262, so it wins whenever u<0.262u < 0.262. The 22 tokens after pieces share the grey tail (0.143 together). Draw 100 adds the observed frequencies underneath. The strip follows the cockpit's TT, top-k and top-p, so filtered tokens shrink to zero width.
Cross-entropy and its gradient1/4
  1. 1
    L=−log⁡py=−zy+log⁡∑jezj\tok{loss}{\mathcal{L}} = -\log \tok{prob}{p_y} = -z_y + \log \sum_{j} e^{z_j}

    y is the token that actually came next in the training text. Only its probability enters the loss.

The gradient in the derivation is what training does to the logits. Turn on Show loss gradient in the cockpit: an amber arrow over every bar points down by pip_i, and a rose arrow over the target points up by 1−py1 - p_y. The arrows use the trained T=1T = 1 softmax, whatever the decoding sliders say, because decoding never enters training. Click any bar to make it the target.

Make the rare idiom pace the token that actually came next, and show the gradient.

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 5.1

Attention Is All You Need, Vaswani et al., 2017, Sec. 3.4

02Scaling laws and compute-optimal training

Pretraining minimises that average loss over a corpus of DD tokens with a model of NN parameters. Both cost compute, and the bill has a simple form.

Where C ≈ 6ND comes from1/4
  1. 1
    forward pass≈2N FLOPs per token\text{forward pass} \approx 2N \ \text{FLOPs per token}

    Each weight takes part in one multiply and one add per token. The attention-score term is left out; it matters only when the context is long relative to the model width.

A fixed budget CC can buy a bigger model trained on fewer tokens or a smaller one trained on more. Hoffmann et al. trained over 400 models and fitted the final loss as an irreducible term plus two power laws (they write the exponents as α and β):

L(N,D)=E+ANNa+BDDb⟹Nopt∝C b/(a+b),Dopt∝C a/(a+b)\tok{loss}{\mathcal{L}}(N, D) = E + \frac{A_N}{N^{a}} + \frac{B_D}{D^{b}} \quad\Longrightarrow\quad N_{\text{opt}} \propto C^{\,b/(a+b)}, \qquad D_{\text{opt}} \propto C^{\,a/(a+b)}
Minimise L subject to C = 6ND. Their fit gives a ≈ 0.34 and b ≈ 0.28, so the two exponents are about 0.45 and 0.55.
Where the exponent b/(a + b) comes from (optional)1/3
  1. 1
    L(N)=E+ANN−a+BD(6C)bN b\mathcal{L}(N) = E + A_N N^{-a} + B_D \Big(\frac{6}{C}\Big)^{b} N^{\,b}

    Fix the budget and substitute D = C/6N: the data term now grows with N, because a bigger model sees fewer tokens.

Loading figure
One fixed budget CC, every model size. Too small (left) and the model cannot fit the data however many tokens it sees; too big (right) and it sees too few tokens. The widget uses the 2024 replication's constants (E 1.82, A 482, B 2085, a 0.348, b 0.366), which reproduce the 20:1 rule. At C=5.8×1023C = 5.8 \times 10^{23} the emerald optimum is a 72B model on 18 tokens per parameter (loss 1.974); Gopher's 280B at the same budget reaches only 1.993. Drag CC: the optimum moves right while D/ND/N stays near 20. Turn on serving cost to see why heavily used models are trained smaller and longer.

All three of the paper's estimation methods put both exponents near 0.5: when compute doubles, parameters and tokens should each grow by about 2\sqrt{2}, so the ratio D/ND/N stays roughly constant. At the scales studied it came out near 20 tokens per parameter, which is why Chinchilla (70B parameters, 1.4T tokens) beat the 280B-parameter Gopher trained with the same compute. In the viewport, a lower loss at this position means more mass on the right bar: if block is the target, going from 1.34 to 1.20 nats means p(block)p(\text{block}) rising from 0.262 to 0.301.

Training Compute-Optimal Large Language Models, Hoffmann et al., 2022, Sec. 3; Table 3

03Temperature: from greedy to uniform

Training only shapes pp. At inference time a decoder decides how to turn pp into one token, and the simplest knob is the temperature T\tok{temperature}{T}, which divides the logits before the softmax:

pi(T)=ezi/T∑jezj/T\tok{prob}{p_i}(\tok{temperature}{T}) = \frac{e^{\tok{logits}{z_i}/\tok{temperature}{T}}}{\sum_j e^{\tok{logits}{z_j}/\tok{temperature}{T}}}
T = 1 is the trained model. The ranking of the tokens never changes, only how peaked the distribution is.
How entropy depends on T1/4
  1. 1
    log⁡pi=ziT−log⁡Z(T),Z(T)=∑jezj/T\log p_i = \frac{z_i}{T} - \log Z(T), \qquad Z(T) = \sum_j e^{z_j/T}

    Take the log of the tempered softmax (natural logs; divide by ln 2 for bits).

Cool the distribution almost to zero: T=0.05T = 0.05.

Heat it up: T=2.5T = 2.5.

Low temperature suits tasks with one right answer: extraction, classification, code. Higher temperature buys variety for brainstorming and creative text, at the price of more draws from the tail. Even at T=0T = 0 some serving stacks are not bit-for-bit reproducible, because batched floating-point arithmetic can flip near-ties between the top two logits.

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 5.3.1

The rest of this lesson is in Pro

  • 04Top-k and nucleus (top-p) sampling
  • 05Fine-tuning: full versus LoRA
  • 06Prompting, RAG or fine-tuning?
  • Python challenge: Temperature, top-k, top-p and a LoRA layer
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What does temperature do in an LLM?
Temperature divides the model's logits before the softmax. Values below 1 sharpen the distribution toward the most likely token, and as temperature approaches 0 sampling becomes greedy decoding. Values above 1 flatten it toward uniform, which adds variety and makes unlikely tokens more frequent. Temperature never changes the ranking of the tokens, only how much probability the top ones get.
What is the difference between top-k and top-p sampling?
Top-k keeps a fixed number of the most likely tokens. Top-p, also called nucleus sampling, keeps the smallest set of most likely tokens whose probabilities add up to at least p, including the token that crosses the threshold. Both renormalise before sampling. Top-p adapts to the model's confidence: it keeps few tokens when one answer dominates and many when the distribution is flat.
How many parameters does LoRA train?
For a weight matrix with d_in inputs and d_out outputs, a rank r LoRA adapter trains r times (d_in plus d_out) parameters, the entries of B (d_out by r) and A (r by d_in). For a 4096 by 4096 layer at rank 8 that is 65,536 parameters, about 0.39 percent of the 16.8 million frozen weights. The update is scaled by alpha over r and can be merged into the weights after training, so inference costs nothing extra.

Next in Transformers and LLMs

4.5Embeddings, vector search and retrieval-augmented generationEmbed a corpus, fire a query into it, and follow nearest-neighbour search through an HNSW graph before the results reach the prompt.

Sources

  • Build a Large Language Model (From Scratch), Sebastian Raschka
  • The Curious Case of Neural Text Degeneration, Holtzman et al., 2020
  • LoRA: Low-Rank Adaptation of Large Language Models, Hu et al., 2021
  • Training Compute-Optimal Large Language Models, Hoffmann et al., 2022
  • Attention Is All You Need, Vaswani et al., 2017