Flash Invariant Point Attention

Andrew Liu (Columbia University) · Axel Elaldi (Flagship Pioneering) · Nicholas Franklin (Flagship Pioneering) · Nathan Russell (Flagship Pioneering) · Gurinder Atwal (Flagship Pioneering) · Yih-En Ban (Flagship Pioneering) · Olivia Viessmann (Flagship Pioneering)

computational costsfactorized reformulationflashattentionflashipagenerative modelsgeometry-aware modelinggpu memoryhardware-efficientinvariant point attentionlinear scalingprotein modelsquadratic complexityrna modelsstructural biologystructural generationtraining lengths

Invariant Point Attention (IPA) is a key algorithm for geometry-aware modeling in structural biology, central to many protein and RNA models. However, its quadratic complexity limits the input sequence length. We introduce FlashIPA, a factorized reformulation of IPA that leverages hardware-efficient FlashAttention to achieve linear scaling in GPU memory and wall-clock time with sequence length. FlashIPA matches or exceeds standard IPA performance while substantially reducing computational costs. FlashIPA extends training to previously unattainable lengths, and we demonstrate this by re-training generative models without length restrictions and generating structures of thousands of residues. FlashIPA is available at https://github.com/flagshippioneering/flash_ipa.