Transformer Architecture: The Positional Encoding

Abstraction: Sinusoidal positional encoding mechanics in transformer models

Key points:

• Transformers lack recurrence so position must be injected via a d-dimensional positional vector added to word embeddings
• Encoding uses paired sine/cosine functions at geometrically decreasing frequencies: ω_k = 1/10000^(2k/d)
• The sinusoidal scheme is analogous to binary counting — each dimension alternates at a different rate
• A linear transformation M (independent of position t) can map PE_t to PE_{t+φ}, enabling the model to learn relative-position attention
• M is equivalent to a rotation matrix, making relative-position reasoning structurally accessible
• Summation (not concatenation) of positional and word embeddings preserves parameter count; residual connections propagate this signal through deep layers

Connections: Transformers · Positional Encoding · Attention Mechanism

Source: https://kazemnejad.com/blog/transformer_architecture_positional_encoding/