On the Emergence of Linear Analogies in Word Embeddings

Dhruva Karkada (University of California, Berkeley) · Yasaman Bahri (Google DeepMind) · Daniel Korchinski (EPFL) · Matthieu Wyart (Swiss Federal Institute of Technology Lausanne)
analogy benchmarkattribute-based interactionsco-occurrence probabilityco-occurrence statisticsdimensionality controleigenvectorsembedding dimensiongenerative modellinear analogy structurematrix factorizationnoise robustnesssemantic attributessemantic similaritytheoretical originword embeddings

Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure