What is nn.Embedding really?

Abstraction: PyTorch embedding layer as efficient lookup-based linear layer equivalent

Key points:

• nn.Embedding is functionally equivalent to a linear layer without a bias term but performs an index lookup instead of matrix-vector multiplication
• Embeddings are more efficient than linear layers for large sparse inputs (e.g., bag-of-words NLP data from CountVectorizer/TfidfVectorizer)
• Embedding weights and linear layer weights are transposes of each other
• The lookup avoids materializing a dense matrix, making it faster and more memory-efficient for sparse data
• After lookup, a sum(0) is required to replicate the linear layer's output
• Insight credited to Jeremy Howard

Connections: Pytorch · Embeddings · Neural Networks · Natural Language Processing

Source: https://medium.com/@gautam.e/what-is-nn-embedding-really-de038baadd24