A New AI Research Introduces EXPHORMER: A Framework For Scaling Graph Transformers While Slashing Costs

Abstraction: Expander-graph sparse attention scales graph transformers to linear complexity

Key points:

• EXPHORMER is a graph transformer framework from University of British Columbia, Google Research, and Alberta Machine Intelligence Institute that achieves linear complexity in the number of nodes and edges
• Uses three attention components: expander graph attention (long-range propagation without full connectivity), global virtual node attention (universal approximation), and local neighborhood attention
• Expander graphs provide desirable mathematical properties — spectral expansion, sparsity, and pseudorandomness — enabling sparse approximation of full attention
• Integrated into the GraphGPS modular framework, EXPHORMER achieves state-of-the-art results on three benchmark datasets despite having fewer parameters than dense transformer alternatives
• Theoretically demonstrated to spectrally approximate full attention with only a small number of layers and to have universal approximation properties
• Handles larger graphs than prior graph transformer architectures while surpassing all other sparse attention mechanisms

Connections: Google Research · Exphormer · Graph Transformers · Graph Neural Networks · Sparse Attention

Source: https://www.marktechpost.com/2023/03/18/a-new-ai-research-introduces-exphormer-a-framework-for-scaling-graph-transformers-while-slashing-costs/