Byte Pair Encoding

concepts · 2 notes linked

Related: Andrej Karpathy · Fast AI · Tiktoken · Sentencepiece · GPT-4 · Tokenization · Large Language Models · Hugging Face

Notes

• How to train a new language model from scratch using Transformers and Tokenizers — End-to-end tutorial training RoBERTa-like LM on Esperanto from scratch
• Let's Build the GPT Tokenizer — A Complete Guide to Tokenization in LLMs — Karpathy tokenizer video as book chapter