The Illustrated Transformer

Abstraction: Visual walkthrough of the Transformer architecture and self-attention

Key points:

• Transformer (from "Attention Is All You Need") outperforms Google NMT and is highly parallelizable, unlike RNNs.
• Each encoder layer has two sub-layers: self-attention and feed-forward network; each with residual connection and layer norm.
• Self-attention computes Query, Key, Value vectors (dim 64) from input embeddings (dim 512); score = dot(Q, K) / sqrt(64), softmaxed, then weighted sum of V.
• Multi-head attention (8 heads in the original) runs self-attention in parallel across multiple representation subspaces, concatenating outputs.
• Positional encoding adds sine/cosine signals to embeddings to inject word-order information; generalizes to unseen sequence lengths.
• Decoder uses masked self-attention (future positions set to -inf) plus encoder-decoder cross-attention to generate output tokens autoregressively.

Connections: Jay Alammar · Transformers · Attention Mechanism · Positional Encoding

Source: http://jalammar.github.io/illustrated-transformer/