Transformers

concepts · 48 notes linked

Related: Large Language Models · Attention Mechanism · Openai · Google · Hugging Face · Positional Encoding · Github · Bert

Notes

• 8 Google Employees Invented Modern AI. Here's the Inside Story — Origin story of "Attention Is All You Need" transformer paper at Google
• A New AI Research Introduces EXPHORMER: A Framework For Scaling Graph Transformers While Slashing Costs — Expander-graph sparse attention scales graph transformers to linear complexity
• A Short Chronology Of Deep Learning For Tabular Data — Chronological survey comparing deep learning versus gradient boosting on tabular data
• A Wave Of Billion-Dollar Language AI Startups Is Coming — 2022 landscape survey of language AI startup ecosystem categories
• Attention in Neural Networks | Towards Data Science — Survey of attention mechanism variants beyond basic encoder-decoder attention
• ChatGPT is 'not particularly innovative,' and 'nothing revolutionary', says Meta's chief AI scientist — Yann LeCun argues ChatGPT is solid engineering not scientific breakthrough
• Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms — Attention as mean-field fixed-point response of Ising spin systems
• DeepMind Paper Provides a Mathematically Precise Overview of Transformer Architectures and Algorithms | Synced — DeepMind formal pseudocode reference for all major transformer architectures
• Efficient transformers: Survey of recent work — Taxonomy and survey of efficient transformer architectures through 2022
• End of an Era at Google DeepMind Hints at New Future for AI — Last "Attention Is All You Need" author departs Google, ending transformer paper era
• Fine-tuning · Hugging Face — Fine-tuning pretrained LLMs with Hugging Face Trainer API
• GitHub - HandsOnLLM/Hands-On-Large-Language-Models: Official code repo for the O'Reilly Book - "Hands-On Large Language Models — Official code repo for illustrated O'Reilly LLM book
• GitHub - badaramoni/wave-field-llm: Wave Field AI — a efficient attention architecture for language models — O(N log N) FFT-based attention replacing quadratic dot-product attention
• GitHub - catalyst-team/dl-course: Deep Learning with Catalyst — Open-source deep learning course using PyTorch and Catalyst framework
• GitHub - huggingface/education-toolkit: Educational materials for universities — Hugging Face free open-source ML education toolkit for universities and workshops
• GitHub - jessevig/bertviz: BertViz: Visualize Attention in Transformer Models — Interactive Jupyter tool visualizing transformer attention weights
• GitHub - sainathadapa/attention-primer-pytorch — PyTorch toy tasks demonstrating attention mechanisms from Vaswani et al.
• Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs — FNet replaces transformer self-attention with Fourier Transform for faster training
• Graph Classification with Transformers — Tutorial fine-tuning Microsoft Graphormer for graph classification via HuggingFace Transformers
• How to train a new language model from scratch using Transformers and Tokenizers — End-to-end tutorial training RoBERTa-like LM on Esperanto from scratch
• Introduction to Graph Machine Learning — Graph neural network fundamentals from representations to transformers
• Once "too scary" to release, GPT-2 gets squeezed into an Excel spreadsheet — GPT-2 fully implemented in Excel spreadsheet for LLM education
• PEGASUS: A State-of-the-Art Model for Abstractive Text Summarization — Google model using gap-sentence pre-training for abstractive summarization
• PyTorch Explained: From Automatic Differentiation to Training Custom Neural Networks | Towards Data Science — PyTorch core abstractions from tensors and autograd to transformer encoder
• Scaling Recommender Transformers to a Billion Parameters | Towards Data Science — ARGUS autoregressive recommender transformer scaled to one billion parameters
• Self-supervised learning is the future of AI — LeCun's blueprint for self-supervised learning over supervised AI
• Self-supervised learning: The plan to make deep learning data-efficient - TechTalks — Yann LeCun's AAAI 2020 roadmap for data-efficient self-supervised learning
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Language-independent subword tokenizer trained from raw sentences
• Test-Time Training (TTT): A New Approach to Sequence Modeling — Hidden state as a learnable model updated at inference
• The Annotated Transformer — Line-by-line annotated implementation of Attention Is All You Need
• The Illustrated Transformer — Visual walkthrough of the Transformer architecture and self-attention
• The Singular Value Decompositions of Transformer Weight Matrices — SVD of GPT-2 weight matrices reveals interpretable semantic directions
• The pathway to Transformers — Technical walkthrough of architectural evolution from RNNs to Transformers
• Transformer Architecture: The Positional Encoding — Sinusoidal positional encoding mechanics in transformer models
• Transformer Explainer: LLM Transformer Model Visually Explained — Interactive visual walkthrough of Transformer/GPT-2 architecture
• Transformers Explained Visually (Part 2): How it works, step-by-step — Step-by-step internal data flow through the Transformer architecture
• Transformers Explained Visually (Part 3): Multi-head Attention, deep dive — Deep dive into multi-head attention mechanics and data dimensions
• Transforming the Retail Industry with Transformers — BERT transformers automating retail product attribute classification in production
• Tsinghua U & Microsoft Propose Fastformer: An Additive Attention Based Transformer With Linear Complexity — Fastformer additive attention transformer achieves linear complexity for long documents
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) — Visual walkthrough of seq2seq encoder-decoder and attention mechanism
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension — Hybrid convolutional-transformer model via knowledge distillation for fast NLU inference
• Welcome to the UvA Deep Learning Tutorials! — University of Amsterdam Jupyter notebook deep learning course covering PyTorch and JAX
• Why humans can't use natural language processing to speak with the animals - Engadget — Why NLP cannot yet decode animal communication systems into human language
• Why is 10000 used as the denominator in Positional Encodings in the Transformer Model? — Rationale for the 10000 base constant in Transformer positional encoding
• Why multi-head self attention works: math, intuitions and 10+1 hidden insights | AI Summer — Mathematical and intuitive deep-dive into multi-head self-attention mechanisms
• Will Transformers Take Over Artificial Intelligence? | Quanta Magazine — Transformers expanding from NLP to vision, generative, and multimodal AI tasks
• Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer) — Autoformer transformer outperforms DLinear baseline for time series forecasting
• elvis (@omarsar0) on X — Stanford CME295 new course on Transformers and LLMs announced