Tsinghua U & Microsoft Propose Fastformer: An Additive Attention Based Transformer With Linear Complexity

Abstraction: Fastformer additive attention transformer achieves linear complexity for long documents

Key points:

• Fastformer replaces standard quadratic self-attention with additive attention: the query matrix is summarized into a global query vector, then interactions with keys/values are modeled via element-wise product, reducing complexity from O(n²) to O(n)
• Benchmarked on five datasets (Amazon, IMDB, MIND, CNN/DailyMail, PubMed) across classification, news recommendation, and summarization tasks; outperformed Longformer, BigBird, Linformer, and Poolingformer in speed
• Fastformer was faster in both training and inference than other linear-complexity transformer variants, making it practically the most efficient architecture tested
• Uses GloVe embeddings with an additive attention network for final pooling; trained with Adam optimizer
• Future work targets pretraining Fastformer-based language models and applications in e-commerce recommendation and CTR prediction

Connections: Microsoft · Tsinghua University · Transformers · Attention Mechanism · Natural Language Processing

Source: https://medium.com/syncedreview/tsinghua-u-microsoft-propose-fastformer-an-additive-attention-based-transformer-with-linear-e6cea7d2feed