Efficient transformers: Survey of recent work

Abstraction: Taxonomy and survey of efficient transformer architectures through 2022

Key points:

• Categorizes efficient transformers into six axes: computational complexity, spectral complexity, robustness, privacy, approximation, and model compression
• Sparse approaches (Longformer, BigBird, Linformer) reduce quadratic O(N²) attention to sub-quadratic via windowed local attention, graph sparsification, or linear projections; Google's Performer uses FAVOR+ kernel approximation
• Spectral methods (FNet, GFNet) replace self-attention with Fourier transforms, running 80% faster than BERT while retaining 92–97% performance
• Vision Transformers (ViT) are more adversarially robust than CNNs but less sensitive to high-frequency image components; combining CNNs with transformers further improves robustness
• Privacy-preserving inference achieved via homomorphic encryption (THE-X method) using polynomial approximations of GELU and SoftMax on BERT-Tiny
• Frozen Pretrained Transformer (FPT) pre-trained only on language data performs comparably to task-specific transformers on protein sequences, numerical computation, and image classification

Connections: Microsoft · Google · Transformers · Attention Mechanism · Model Compression

Source: https://medium.com/data-science-at-microsoft/efficient-transformers-survey-of-recent-work-75022cddc86a