Skip to content
LearnMLByDoing
Free

4.2 Tokenisation and embedding spaces

30 minProintermediateRaschkaVaswaniBPERoPED2LCompendium

A language model never sees text: a tokenizer cuts it into pieces from a fixed vocabulary, an embedding table turns each piece into a vector, and a position encoding tells attention where each vector sits.

01Why tokens: characters, words and subwords

A transformer computes with vectors, so text has to become a sequence of integers first. A tokenizer cuts the text into pieces from a fixed vocabulary of V\tok{vocab}{V} entries and replaces each piece by its id. Everything downstream is counted in those pieces: the context window, the price of a request, and the N2N^2 cost of attention from lesson 4.1.

The size of the pieces is a trade-off. Characters give a tiny vocabulary and never meet an unknown symbol, but sequences are long and each token carries little meaning. Words give short sequences, but the vocabulary explodes with inflections, names and typos, and any word missing from it becomes an unknown token. Subwords sit in between: frequent words stay whole, rare words are split into reusable pieces, and VV is a number you choose (50,257 for GPT-2).

T⏟sequence length≈characterscharacters per token,V⋅dmodel⏟embedding table parameters\underbrace{\tok{token_count}{T}}_{\text{sequence length}} \approx \frac{\text{characters}}{\text{characters per token}}, \qquad \underbrace{\tok{vocab}{V}\cdot d_{\text{model}}}_{\text{embedding table}} \ \text{parameters}
A larger vocabulary means fewer, longer tokens but a bigger embedding table and output softmax.

The viewport replays byte-pair encoding on a toy corpus of 12 word types with their counts: 52 words, 292 characters, an alphabet of 14 letters. At merge 0 every character is a token, so T=292T = 292 and V=14V = 14. Each merge fuses one pair of tokens everywhere it occurs, and after 30 merges every word is a single token: T=52T = 52, V=44V = 44, one token per word. Everything in between is subword tokenization; the bar chart on the right is TT after each merge.

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 2.2–2.4

Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016, Sec. 1 and 3

02Byte-pair encoding: learning the merges

Byte-pair encoding was a data-compression method before Sennrich et al. used it to build subword vocabularies for translation. Training starts from single characters and repeats one rule: count every adjacent pair of symbols, weighting each word by how often it occurs, and merge the most frequent pair everywhere.

cab=∑wfw⋅#ab(w),(a∗,b∗)=arg⁡max⁡(a,b)cab\tok{merge}{c_{ab}} = \sum_{w} f_w \cdot \#_{ab}(w), \qquad (a^{*}, b^{*}) = \arg\max_{(a,b)} \tok{merge}{c_{ab}}
f_w is the count of word w and #_ab(w) the number of times symbol a is followed by symbol b in its current segmentation.
Loading figure
The count table behind each merge, for the board's current KK. The amber bar is the next merge; a tie mark means another pair has the same count and alphabetical order decides. next merge advances the 3D board by one merge (and pauses its replay).
Why greedy merges compress1/4
  1. 1
    T=∑wfw ∣sw∣\tok{token_count}{T} = \sum_{w} f_w \,|s_w|

    The corpus length is the number of symbols in each word's segmentation s_w, weighted by the word's count.

Watch the replay (the bar for merge kk in the chart on the right is TkT_k). Merge 1 is e + s: it and s + t both occur 23 times, and the tie goes to the smaller pair. Merge 2 builds est, merge 3 er, merges 4 and 5 ld and old. After the 10 merges set in the viewport the counts have fallen from 23 to 8, TT has dropped from 292 to 147 (the ten counts sum to 145) and VV is 24: about 2 characters per token, 2.83 tokens per word. Hover cab\tok{merge}{c_{ab}} to light up the next merge, or any tile to read its token id.

Replay the first 12 merges and watch the last one.

Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016, Sec. 3.2, Algorithm 1

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 2.5

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 15.6, byte pair encoding

03Encoding new text, bytes, and why models miscount letters

To encode a word the tokenizer starts from its characters and replays the merges in the order they were learned. The two rows under the corpus are words it never saw. boldest ends as bold · est: merge 2 made est and merge 24 made bold, so the unseen word reuses two pieces. slowly ends as s · low · l · y, and its y turns rose because the letter is not in the 14-character alphabet. A character-level tokenizer has to emit an unknown token there.

Loading figure
Your own words through the same 30 merges. At K=30K = 30 the boldest walker becomes t · h · e, bold · est, w · al · k · er: 9 tokens for 16 letters: the and walk never occur in the toy corpus, and h and k are not even in its alphabet (rose). Turn on byte-level fallback and type a letter outside the alphabet, such as the y of slowly or the é of café (two bytes).

Stop the board at 23 merges, one before bold exists.

encode⁡(w)=merge⁡K∘⋯∘merge⁡2∘merge⁡1(chars⁡(w))\operatorname{encode}(w) = \operatorname{merge}_{K} \circ \cdots \circ \operatorname{merge}_{2} \circ \operatorname{merge}_{1}\big(\operatorname{chars}(w)\big)
Rank order matters: with merges [(b, c), (a, b)], abc encodes as a · bc, never ab · c.

Byte-level BPE (GPT-2 and most models since) removes unknown tokens altogether: the base alphabet is the 256 possible byte values of UTF-8, so any string is encodable, and merges are learned on bytes. GPT-2's 50,257 entries are 256 bytes, 50,000 merges and one end-of-text token. A leading space is part of the token, so the token for bold after a space is a different id from bold at the start of a text, and an accented letter or an emoji costs two to four bytes before any merge.

token ids[,]8,192 values

Integers in [0, V). After 30 merges warmest is a single id, 30, so a model reading it never receives the letters w, a, r, m, e, s, t.

Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 2.5

Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016, Sec. 3.2

The rest of this lesson is in Pro

  • 04Embedding tables: a lookup is a matrix product
  • 05Geometry of meaning: cosine, analogies and their limits
  • 06Position: sinusoidal encodings and RoPE
  • 07Practical notes: token budgets, fairness and embedding models
  • Python challenge: A BPE tokenizer and rotary position embedding
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is byte pair encoding (BPE) tokenization?
Byte pair encoding builds a subword vocabulary from data. It starts with single characters or bytes, counts how often each adjacent pair occurs in the training corpus, merges the most frequent pair into a new token, and repeats until the vocabulary reaches the chosen size. The ordered list of merges is the tokenizer: new text is encoded by replaying the merges in the same order, so frequent words become one token and rare words are split into known pieces.
Why can't language models count the letters in a word like strawberry?
The model never receives letters. The tokenizer turns strawberry into one or a few multi-character tokens, and the model sees only their ids and the vectors behind them. Knowing how many r's a token contains requires having learned the spelling of that token from training text, which is unreliable. Splitting the word into single characters first, or asking the model to spell it out before counting, usually helps.
What is rotary position embedding (RoPE)?
RoPE encodes position by rotating the query and key vectors instead of adding a position vector to the embeddings. The dimensions are grouped into pairs, and pair i is rotated by the position times a fixed frequency that falls from 1 radian per position for the first pair to very slow rotations for the last ones. Because both vectors are rotated, their dot product depends only on how far apart the two tokens are, which gives attention relative position information without any learned position parameters.

Next in Transformers and LLMs

4.3Multi-head attention and the transformer blockHeads as parallel subspaces, residual streams, pre-norm and the MLP that stores most of a model's facts.

Sources

  • Build a Large Language Model (From Scratch), Sebastian Raschka
  • Attention Is All You Need, Vaswani et al., 2017
  • Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016
  • RoFormer: Enhanced Transformer with Rotary Position Embedding, Su et al., 2021
  • Dive into Deep Learning, Zhang, Lipton, Li & Smola
  • Maths, CS & AI Compendium, Henry Ndubuaku