Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms

Abstraction: Attention as mean-field fixed-point response of Ising spin systems

Key points:

• Softmax attention implements a single naive mean-field update step of a vector Ising spin model without an Onsager self-correction term
• The feed-forward layer in transformers is reinterpreted as amortizing the Onsager self-correction term from Thouless–Anderson–Palmer (TAP) mean-field theory
• Residual connections correspond to source/magnetic-field terms that inject input data into the mean-field equations
• Adaptive TAP equations are solved iteratively as a deep equilibrium model (DEQ) fixed-point; cost is O(N³d³) for the explicit variant
• A neural "fast" variant replaces expensive cavity variance computation with a learned network f_θ, recovering the vanilla transformer structure
• Achieves ~99.1% MNIST test accuracy with only 26k parameters using a DEQ attention layer
• MLP-Mixer, FNet, and related architectures are all recast as different parametrizations of mean-field interaction terms

Connections: Matthias Bal · Attention Mechanisms · Mean Field Theory · Transformers · Deep Equilibrium Models

Source: https://mcbal.github.io/post/deep-implicit-attention-a-mean-field-theory-perspective-on-attention-mechanisms/