Tiktoken

entities · 1 notes linked

Related: Andrej Karpathy · Fast AI · Sentencepiece · GPT-4 · Tokenization · Byte Pair Encoding · Large Language Models

Notes

• Let's Build the GPT Tokenizer — A Complete Guide to Tokenization in LLMs — Karpathy tokenizer video as book chapter