Attention in Neural Networks | Towards Data Science

Abstraction: Survey of attention mechanism variants beyond basic encoder-decoder attention

Key points:

• Global attention uses all encoder hidden states for context; local attention attends to a 2D+1 window around a predicted alignment position p (monotonic or predictive)
• Soft attention computes weighted sum of all hidden states; hard attention selects a single state (non-differentiable, requires complex techniques like REINFORCE)
• ABCNN (Attention-Based CNN) introduces attention for sentence-pair tasks in three variants: ABCNN-1 (before convolution), ABCNN-2 (re-weights pooling), ABCNN-3 (both)
• Decomposable Attention (Parikh et al.) decomposes NLI into word-level sub-problems using only attention and feed-forward networks; word order is ignored
• Area Attention generalizes attention to groups of adjacent items; key vector = mean of hidden states in area, value = sum; supports richer key representation using mean, std-dev, and shape vectors

Connections: Attention Mechanisms · Neural Networks · Transformers

Source: https://towardsdatascience.com/attention-in-neural-networks-e66920838742