Let's Build the GPT Tokenizer — A Complete Guide to Tokenization in LLMs

tokenizationbyte-pair-encodinggpttiktokensentencepieceunicode

Abstraction: Karpathy tokenizer video as book chapter

Key points:

Text version of Karpathy's 2h13m tokenizer video, produced via fast.ai's Solveit platform (~10x faster than manual); walks through building a GPT-4-style tokenizer from scratch.
Tokenization is the root of many LLM quirks: bad spelling, string reversal, arithmetic, non-English performance, GPT-2's Python struggles, <|endoftext|> halting, YAML-over-JSON preference, "SolidGoldMagikarp."
Foundations: strings are Unicode code points (~150k, unstable standard); UTF-8 (1–4 bytes, ASCII-compatible) preferred over UTF-16/UTF-32; naive bytes give only 256-token vocab and overly long sequences.
BPE algorithm iteratively merges the most frequent byte pair into a new token; GPT-2 used vocab 50,257 / context 1,024; GPT-4 (cl100k) ~100k vocab. Regex pre-tokenization prevents merges across letter/number/punctuation/whitespace categories; GPT-4 pattern adds case-insensitive contractions and 2+ digit number rule.
tiktoken (OpenAI, inference-only) does bytes→BPE; SentencePiece (Llama/Mistral) runs BPE on code points with byte-fallback for rare chars. Exercise via karpathy/minbpe reproduces cl100k exactly.

Connections: Andrej Karpathy · Fast AI · Tiktoken · Sentencepiece · GPT-4 · Tokenization · Byte Pair Encoding · Large Language Models