On the Emergence of Linear Analogies in Word Embeddings
analogy benchmarkattribute-based interactionsco-occurrence probabilityco-occurrence statisticsdimensionality controleigenvectorsembedding dimensiongenerative modellinear analogy structurematrix factorizationnoise robustnesssemantic attributessemantic similaritytheoretical originword embeddings
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure