Improving language models by retrieving from trillions of tokens

Abstraction: RETRO retrieval-augmented LM outperforms models 25x larger on language modeling

Key points:

• DeepMind's RETRO (Retrieval-Enhanced Transformer) performs nearest-neighbor search over a training database for each text passage, retrieving similar sequences and their continuations to condition generation
• Architecture interleaves standard self-attention at the document level with cross-attention over retrieved neighbors at the passage level, producing more factual and topically focused continuations
• A 7.5B parameter RETRO model outperforms the 175B Jurassic-1 on 10 of 16 Pile benchmark datasets and outperforms the 280B Gopher on 9 of 16 — roughly 25x and 37x parameter efficiency respectively
• Retrieval improves interpretability: predictions can be traced to retrieved neighbors, and the retrieval database can be directly edited to improve safety without retraining
• Evaluated on the Pile (standard language modeling benchmark); dataset scale is described as "trillions of tokens" accessible via the external database

Connections: Deepmind · Retro · Retrieval Augmented Generation · Large Language Models

Source: https://www.deepmind.com/blog/improving-language-models-by-retrieving-from-trillions-of-tokens