4.2 Tokenisation and embedding spaces
A language model never sees text: a tokenizer cuts it into pieces from a fixed vocabulary, an embedding table turns each piece into a vector, and a position encoding tells attention where each vector sits.
01Why tokens: characters, words and subwords
A transformer computes with vectors, so text has to become a sequence of integers first. A tokenizer cuts the text into pieces from a fixed vocabulary of entries and replaces each piece by its id. Everything downstream is counted in those pieces: the context window, the price of a request, and the cost of attention from lesson 4.1.
The size of the pieces is a trade-off. Characters give a tiny vocabulary and never meet an unknown symbol, but sequences are long and each token carries little meaning. Words give short sequences, but the vocabulary explodes with inflections, names and typos, and any word missing from it becomes an unknown token. Subwords sit in between: frequent words stay whole, rare words are split into reusable pieces, and is a number you choose (50,257 for GPT-2).
The viewport replays byte-pair encoding on a toy corpus of 12 word types with their counts: 52 words, 292 characters, an alphabet of 14 letters. At merge 0 every character is a token, so and . Each merge fuses one pair of tokens everywhere it occurs, and after 30 merges every word is a single token: , , one token per word. Everything in between is subword tokenization; the bar chart on the right is after each merge.
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 2.2–2.4
Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016, Sec. 1 and 3
02Byte-pair encoding: learning the merges
Byte-pair encoding was a data-compression method before Sennrich et al. used it to build subword vocabularies for translation. Training starts from single characters and repeats one rule: count every adjacent pair of symbols, weighting each word by how often it occurs, and merge the most frequent pair everywhere.
- 1
The corpus length is the number of symbols in each word's segmentation s_w, weighted by the word's count.
Watch the replay (the bar for merge in the chart on the right is ). Merge 1 is e + s: it and s + t both occur 23 times, and the tie goes to the smaller pair. Merge 2 builds est, merge 3 er, merges 4 and 5 ld and old. After the 10 merges set in the viewport the counts have fallen from 23 to 8, has dropped from 292 to 147 (the ten counts sum to 145) and is 24: about 2 characters per token, 2.83 tokens per word. Hover to light up the next merge, or any tile to read its token id.
Replay the first 12 merges and watch the last one.
Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016, Sec. 3.2, Algorithm 1
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 2.5
Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 15.6, byte pair encoding
03Encoding new text, bytes, and why models miscount letters
To encode a word the tokenizer starts from its characters and replays the merges in the order they were learned. The two rows under the corpus are words it never saw. boldest ends as bold · est: merge 2 made est and merge 24 made bold, so the unseen word reuses two pieces. slowly ends as s · low · l · y, and its y turns rose because the letter is not in the 14-character alphabet. A character-level tokenizer has to emit an unknown token there.
Stop the board at 23 merges, one before bold exists.
Byte-level BPE (GPT-2 and most models since) removes unknown tokens altogether: the base alphabet is the 256 possible byte values of UTF-8, so any string is encodable, and merges are learned on bytes. GPT-2's 50,257 entries are 256 bytes, 50,000 merges and one end-of-text token. A leading space is part of the token, so the token for bold after a space is a different id from bold at the start of a text, and an accented letter or an emoji costs two to four bytes before any merge.
Integers in [0, V). After 30 merges warmest is a single id, 30, so a model reading it never receives the letters w, a, r, m, e, s, t.
Build a Large Language Model (From Scratch), Sebastian Raschka, Ch. 2.5
Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016, Sec. 3.2
The rest of this lesson is in Pro
- 04Embedding tables: a lookup is a matrix product
- 05Geometry of meaning: cosine, analogies and their limits
- 06Position: sinusoidal encodings and RoPE
- 07Practical notes: token budgets, fairness and embedding models
- Python challenge: A BPE tokenizer and rotary position embedding
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is byte pair encoding (BPE) tokenization?
- Byte pair encoding builds a subword vocabulary from data. It starts with single characters or bytes, counts how often each adjacent pair occurs in the training corpus, merges the most frequent pair into a new token, and repeats until the vocabulary reaches the chosen size. The ordered list of merges is the tokenizer: new text is encoded by replaying the merges in the same order, so frequent words become one token and rare words are split into known pieces.
- Why can't language models count the letters in a word like strawberry?
- The model never receives letters. The tokenizer turns strawberry into one or a few multi-character tokens, and the model sees only their ids and the vectors behind them. Knowing how many r's a token contains requires having learned the spelling of that token from training text, which is unreliable. Splitting the word into single characters first, or asking the model to spell it out before counting, usually helps.
- What is rotary position embedding (RoPE)?
- RoPE encodes position by rotating the query and key vectors instead of adding a position vector to the embeddings. The dimensions are grouped into pairs, and pair i is rotated by the position times a fixed frequency that falls from 1 radian per position for the first pair to very slow rotations for the last ones. Because both vectors are rotated, their dot product depends only on how far apart the two tokens are, which gives attention relative position information without any learned position parameters.
Next in Transformers and LLMs
4.3Multi-head attention and the transformer blockHeads as parallel subspaces, residual streams, pre-norm and the MLP that stores most of a model's facts.Sources
- Build a Large Language Model (From Scratch), Sebastian Raschka
- Attention Is All You Need, Vaswani et al., 2017
- Neural Machine Translation of Rare Words with Subword Units, Sennrich, Haddow & Birch, 2016
- RoFormer: Enhanced Transformer with Rotary Position Embedding, Su et al., 2021
- Dive into Deep Learning, Zhang, Lipton, Li & Smola
- Maths, CS & AI Compendium, Henry Ndubuaku