Tokenization

concepts · 5 notes linked

Related: Large Language Models · GPT-2 · Transformers · Andrej Karpathy · Openai · Nltk · Natural Language Processing · Text Classification

Notes

• Let's Build the GPT Tokenizer — A Complete Guide to Tokenization in LLMs — Karpathy tokenizer video as book chapter
• NLTK — Python Natural Language Toolkit for human language processing
• Once "too scary" to release, GPT-2 gets squeezed into an Excel spreadsheet — GPT-2 fully implemented in Excel spreadsheet for LLM education
• Transformer Explainer: LLM Transformer Model Visually Explained — Interactive visual walkthrough of Transformer/GPT-2 architecture
• Word embeddings visualization from 'Build a Large Language Model (From Scratch)' — Animated 2D visualization of LLM token-to-vector embedding training