Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

Abstraction: Rationale for the 10000 base constant in Transformer positional encoding

Key points:

• n=10000 ensures cosine distance between position vectors is a monotonically increasing function of position index; smaller n values break this property
• With n=10000, the full sinusoidal cycle spans >50,000 positions — far more than typical sequence lengths (<1000 words)
• For short sentences (15-20 words), roughly half the channels (index i>256 in 512-dim encoding) are near-constant, providing no positional information; this may be intentional
• Adding positional encoding (rather than concatenating) can "corrupt" embedding information; the large n mitigates distortion at typical sequence lengths
• The 1st sinusoid pair uses sin(pos)/cos(pos) directly; uniqueness relies on the irrationality of π for long sequences

Connections: Transformers · Positional Encoding

Source: https://datascience.stackexchange.com/questions/82451/why-is-10000-used-as-the-denominator-in-positional-encodings-in-the-transformer