A Brief Overview of Attention Mechanism
attentionseq2seqrnnnlpneural-machine-translation
Abstraction: Encoder-decoder attention as context vector over all hidden states
Key points:
- Vanilla RNN encoder-decoder compresses all source information into one fixed-length hidden state, causing information loss for long sentences
- Attention adds a context vector computed as a weighted sum of all encoder hidden states, with weights from a softmax over alignment scores
- Two scoring styles compared: Bahdanau (additive, W1 and W2) and Luong (multiplicative, single W matrix)
- Context vector is computed freshly for each decoder output step, giving a 2D attention matrix of size (target_len × source_len)
- Attention allows the decoder to "zoom in" on relevant source positions rather than relying on a single summary state
- Paper "Attention Is All You Need" (Vaswani et al., 2017) is cited as the culmination of this line of work
Connections: Synced · Attention Mechanisms · Sequence To Sequence · Neural Machine Translation
Source: https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129