Attention Mechanism

concepts · 17 notes linked

Related: Transformers · Large Language Models · Positional Encoding · Google · Openai · Microsoft · Github · Cohere

Notes

• 8 Google Employees Invented Modern AI. Here's the Inside Story — Origin story of "Attention Is All You Need" transformer paper at Google
• A Step-by-Step Guide to the Microsoft News Recommendation Competition | Towards Data Science — Walkthrough of NRMS algorithm and MIND competition submission process
• Efficient transformers: Survey of recent work — Taxonomy and survey of efficient transformer architectures through 2022
• End of an Era at Google DeepMind Hints at New Future for AI — Last "Attention Is All You Need" author departs Google, ending transformer paper era
• GitHub - badaramoni/wave-field-llm: Wave Field AI — a efficient attention architecture for language models — O(N log N) FFT-based attention replacing quadratic dot-product attention
• GitHub - jessevig/bertviz: BertViz: Visualize Attention in Transformer Models — Interactive Jupyter tool visualizing transformer attention weights
• GitHub - sainathadapa/attention-primer-pytorch — PyTorch toy tasks demonstrating attention mechanisms from Vaswani et al.
• Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs — FNet replaces transformer self-attention with Fourier Transform for faster training
• The Annotated Transformer — Line-by-line annotated implementation of Attention Is All You Need
• The Illustrated Transformer — Visual walkthrough of the Transformer architecture and self-attention
• The pathway to Transformers — Technical walkthrough of architectural evolution from RNNs to Transformers
• Transformer Architecture: The Positional Encoding — Sinusoidal positional encoding mechanics in transformer models
• Transformers Explained Visually (Part 2): How it works, step-by-step — Step-by-step internal data flow through the Transformer architecture
• Transformers Explained Visually (Part 3): Multi-head Attention, deep dive — Deep dive into multi-head attention mechanics and data dimensions
• Tsinghua U & Microsoft Propose Fastformer: An Additive Attention Based Transformer With Linear Complexity — Fastformer additive attention transformer achieves linear complexity for long documents
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) — Visual walkthrough of seq2seq encoder-decoder and attention mechanism
• Why multi-head self attention works: math, intuitions and 10+1 hidden insights | AI Summer — Mathematical and intuitive deep-dive into multi-head self-attention mechanisms