Positional Encoding

concepts · 6 notes linked

Related: Transformers · Attention Mechanism · Pytorch · Harvard NLP · Opennmt · Jay Alammar · Github · Self Attention

Notes

• GitHub - sainathadapa/attention-primer-pytorch — PyTorch toy tasks demonstrating attention mechanisms from Vaswani et al.
• The Annotated Transformer — Line-by-line annotated implementation of Attention Is All You Need
• The Illustrated Transformer — Visual walkthrough of the Transformer architecture and self-attention
• Transformer Architecture: The Positional Encoding — Sinusoidal positional encoding mechanics in transformer models
• Transformers Explained Visually (Part 2): How it works, step-by-step — Step-by-step internal data flow through the Transformer architecture
• Why is 10000 used as the denominator in Positional Encodings in the Transformer Model? — Rationale for the 10000 base constant in Transformer positional encoding