Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers

Morris Yau (Massachusetts Institute of Technology) · Kazuki Irie (Harvard University) · Samuel J Gershman (Harvard University)

dynamic synaptic modulationfast weight memorygeneral language modelinghybrid memory architectureskey-value memorylinear transformersmemory system blendingneural memory systemspartially observable environmentsquadratic transformersreinforcement learning retrieval tasks sequence processing softmax attentionsynthetic algorithmic tasks

We develop hybrid memory architectures for general-purpose sequence processing neural networks, that combine key-value memory using softmax attention (KV-memory) with fast weight memory through dynamic synaptic modulation (FW-memory)