Normalization in Attention Dynamics

Yury Polyanskiy (Massachusetts Institute of Technology) · Nikita Karagodin (Massachusetts Institute of Technology) · Shu Ge (Massachusetts Institute of Technology) · Philippe Rigollet (MIT)
clustering dynamicsdeep transformerseffective choiceinteracting particlesln-scalingmix-lnngptnormalization schemesperi-lnpost-lnpre-lnprincipled basisrepresentation collapsespeed regulationtoken representationsunified analysis

We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes