The pathway to Transformers

transformersattentionencoder-decoderbertgptnlparchitecture

Abstraction: Technical walkthrough of architectural evolution from RNNs to Transformers

Key points:

Encoder-decoder architecture introduced by Cho et al. (2014) for statistical machine translation using RNNs: encoder compresses input to a fixed context vector, decoder generates output sequence from it
Key limitation of basic encoder-decoder: all information compressed into a single final hidden state, causing loss of information for long sequences
Bahdanau attention (2014) fixes this by using bidirectional RNNs and computing a different weighted context vector per decoder output step, letting the decoder selectively attend to relevant encoder states
Self-attention (Lin et al.) moves attention into the encoder step itself, producing a context matrix that captures multiple parts of the input simultaneously
Transformer (Vaswani et al., 2017) replaces sequential RNN computation with Scaled Dot-Product Multi-Head Attention, enabling full parallelization; uses Q/K/V projections learned during training
Transformer encoder = N layers of (multi-head self-attention + FFN + residual + norm); decoder adds cross-attention to encoder output and uses masked self-attention; BERT = encoder-only, GPT = decoder-only

Connections: Google · Openai · Transformers · Attention Mechanism · Encoder Decoder