Attention Mechanisms

concepts · 3 notes linked

Related: Transformers · Synced · Sequence To Sequence · Neural Machine Translation · Matthias Bal · Mean Field Theory · Deep Equilibrium Models · Neural Networks

Notes

• A Brief Overview of Attention Mechanism — Encoder-decoder attention as context vector over all hidden states
• Attention in Neural Networks | Towards Data Science — Survey of attention mechanism variants beyond basic encoder-decoder attention
• Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms — Attention as mean-field fixed-point response of Ising spin systems