GitHub - sainathadapa/attention-primer-pytorch

Abstraction: PyTorch toy tasks demonstrating attention mechanisms from Vaswani et al.

Key points:

• Fork of greentfrapp/attention-primer replacing TensorFlow with PyTorch; no RNNs or CNNs used in any model
• Five progressive tasks: counting letters (scaled dot-product attention), difference (self-attention), signal (positional encodings), signal2 (multi-head attention), translation (full Transformer on IWSLT 2016 de-en dataset)
• Demonstrates Scaled Dot-Product Attention from Vaswani et al. 2017 (Section 3.2.1)
• Positional encoding task shows L2-norm of first-position encoding is 0.816 vs ~0.18 for remaining positions
• Self-attention weights reveal output steps attending to themselves (narcissistic) unless dependencies require cross-attention
• Translation task on IWSLT 2016 de-en, much smaller than WMT but sufficient to demonstrate Transformer

Connections: Github · Pytorch · Transformers · Attention Mechanism · Self Attention · Positional Encoding

Source: https://github.com/sainathadapa/attention-primer-pytorch