Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

Abstraction: Visual walkthrough of seq2seq encoder-decoder and attention mechanism

Key points:

• Seq2seq encoder-decoder: encoder RNN compresses full input sequence into a fixed-size context vector; decoder RNN generates output tokens from it
• The fixed context vector is a bottleneck for long sentences; attention (Bahdanau 2014, Luong 2015) addresses this by passing all encoder hidden states to the decoder
• Attention decoder scores each encoder hidden state at every decoding step, softmax-weights them, and builds a per-step context vector
• Word embeddings (200–300 dimensions typical; 4 used in examples) represent tokens as continuous vectors before RNN ingestion
• Context vector size equals number of encoder RNN hidden units—typically 256, 512, or 1024 in production
• Google Translate adopted seq2seq in production in late 2016; paper by Jay Alammar widely used as introductory reference

Connections: Jay Alammar · Seq2seq · Attention Mechanism · Neural Machine Translation

Source: https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/