8 Google Employees Invented Modern AI. Here's the Inside Story

transformersattention-mechanismgooglehistory-of-aideep-learning

Abstraction: Origin story of "Attention Is All You Need" transformer paper at Google

Key points:

"Attention Is All You Need" (2017) authored by 8 Google researchers (equal contributors, random order); Noam Shazeer, Jakob Uszkoreit, Ashish Vaswani, Illia Polosukhin, Niki Parmar, Llion Jones, Lukasz Kaiser, Aidan Gomez
Self-attention concept originated with Jakob Uszkoreit ~2014 as a fix for recurrent neural networks' inability to scale to long sequences
Shazeer's implementation ("magic/alchemy") broke the team's plateau and produced record BLEU translation scores; "Big" model trained in 3.5 days
All 8 authors have since left Google; combined startups (Cohere, Character AI, Adept/Essential AI, Sakana AI, Inceptive) are worth billions
OpenAI moved faster: Ilya Sutskever directed Alec Radford to build on the paper, producing the first GPT series; Google could have had GPT-3 by 2019-2020
Transformer name chosen by Uszkoreit from childhood Hasbro toys; title "Attention Is All You Need" suggested by Llion Jones referencing The Beatles

Connections: Google · Openai · Noam Shazeer · Cohere · Transformers · Attention Mechanism · Large Language Models