transformer architectures
A type of neural network architecture that relies on self-attention mechanisms to process sequential data, widely used in natural language processing for tasks such as translation and text generation.
- A unified framework for establishing the universal approximation of transformer-type architectures
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
- Architectural and Inferential Inductive Biases for Exchangeable Sequence Modeling
- Attention on the Sphere
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
- Fixed-Point RNNs: Interpolating from Diagonal to Dense
- Generalized and Invariant Single-Neuron In-Vivo Activity Representation Learning
- Impact of Layer Norm on Memorization and Generalization in Transformers
- In-Context Compositional Learning vis Sparse Coding Transformer
- Learning in Compact Spaces with Approximately Normalized Transformer
- On the Robustness of Transformers against Context Hijacking for Linear Classification
- QSCA: Quantization with Self-Compensating Auxiliary for Monocular Depth Estimation
- Revitalizing SVD for Global Covariance Pooling: Halley’s Method to Overcome Over-Flattening
- SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation
- Spectral Conditioning of Attention Improves Transformer Performance
- TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
- Understanding Softmax Attention Layers:\\ Exact Mean-Field Analysis on a Toy Problem