Transformers: Attention is All You Need

Revolutionizing Sequence Modeling

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Key Innovation: Self-attention mechanism enables parallel processing of sequences

Paper: "Attention is All You Need" (Vaswani et al., 2017)
Problem Solved: Sequential bottleneck in RNNs and LSTMs
Key Benefit: Parallelizable training with long-range dependencies
Impact: Foundation for GPT, BERT, and modern large language models

            Revolutionary Insight: You don't need recurrence or convolution—attention alone is sufficient for sequence modeling
        

Problems with RNNs and CNNs

Why We Needed Something Better

RNN/LSTM Limitations:

Sequential Processing: Can't parallelize—must process tokens one by one
Vanishing Gradients: Difficulty learning long-range dependencies
Memory Bottleneck: Fixed hidden state size limits information flow
Training Time: Sequential nature makes training slow

CNN Limitations for Sequences:

Fixed Receptive Field: Need many layers for long-range dependencies
Position Sensitivity: Not naturally suited for variable-length sequences
Locality Bias: Assumes local patterns, but language has global structure

Transformer Solution: Direct connections between any two positions with self-attention

The Attention Mechanism

Query, Key, Value Paradigm

Intuition: Like searching a database with query-key matching

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Query (Q): "What am I looking for?"
- Representation of the current position/token
Key (K): "What do I have available?"
- Representations of all positions that can be attended to
Value (V): "What information do I get?"
- Actual information to be retrieved from attended positions

Attention Weight = Query · Key
Output = Σ (Attention Weight × Value)

            Scaling Factor $\sqrt{d_k}$: Prevents softmax from saturating when $d_k$ is large
        

Self-Attention

Tokens Attending to Themselves

Key Idea: Each token in a sequence attends to all other tokens (including itself)

Input: "The cat sat on the mat"

Self-attention allows:
• "cat" to attend to "sat" (subject-verb relationship)
• "sat" to attend to "mat" (verb-object relationship)
• "the" to attend to "cat" and "mat" (determiner-noun)
• All tokens to build contextual representations

$$Q = XW_Q, \quad K = XW_K, \quad V = XW_V$$ $$\text{where } X \text{ is the input sequence}$$

Input: Sequence of token embeddings $X \in \mathbb{R}^{n \times d}$
Linear Projections: $W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}$
Attention Matrix: $A_{ij}$ = how much token $i$ attends to token $j$
Output: Contextual representations for each token

Complexity: $O(n^2 \cdot d)$ where $n$ is sequence length, $d$ is dimension

Multi-Head Attention

Multiple Attention Mechanisms in Parallel

Motivation: Different heads can capture different types of relationships

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$ $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$$

Why Multiple Heads?

Head 1: Syntactic relationships (subject-verb, noun-adjective)
Head 2: Semantic relationships (word similarity, coreference)
Head 3: Positional relationships (nearby words, discourse)
Head 4: Long-range dependencies (sentence-level structure)

Parallel Processing: All heads computed simultaneously
Different Subspaces: Each head learns different representation
Concatenation: Combine all head outputs
Final Projection: $W^O$ learns to combine head information

Typical Configuration: 8-16 heads, $d_{model} = 512$, $d_k = d_v = 64$

The Transformer Block

Complete Layer Architecture

Input
↓
Multi-Head Self-Attention
↓
Add & Norm (Residual Connection + Layer Norm)
↓
Feed-Forward Network
↓
Add & Norm (Residual Connection + Layer Norm)
↓
Output

Key Components:

Multi-Head Attention: Self-attention with multiple heads
Residual Connections: $\text{output} = \text{sublayer}(\text{input}) + \text{input}$
Layer Normalization: Stabilizes training, applied after residual
Feed-Forward Network: Position-wise fully connected layers

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$ $$\text{LayerNorm}(x + \text{Sublayer}(x))$$

Stacking: Typically 6-12 transformer blocks for encoder, 6-12 for decoder

Positional Encoding

Adding Position Information

Problem: Self-attention is permutation invariant—no inherent position information

Solution: Add positional encodings to input embeddings

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

Sinusoidal Encoding Properties:

Unique: Each position gets a unique encoding
Relative: Model can learn relative positions
Extrapolation: Can handle longer sequences than seen in training
Smooth: Similar positions have similar encodings

Final Input = Token Embedding + Positional Encoding
Both have same dimension $d_{model}$, so they can be added

Alternative: Learned positional embeddings (used in BERT, GPT)

Encoder-Decoder Architecture

Original Transformer Design

Use Case: Machine translation (e.g., English → French)

Encoder Stack (6 layers):
• Processes input sequence (English)
• Self-attention looks at all input tokens
• Produces contextual representations

Decoder Stack (6 layers):
• Generates output sequence (French)
• Masked self-attention (can't see future)
• Cross-attention to encoder output
• Autoregressive generation

Component	Encoder	Decoder
Self-Attention	Full attention (bidirectional)	Masked attention (causal)
Cross-Attention	None	Attends to encoder output
Input	Source sequence	Target sequence (shifted)
Output	Contextual representations	Next token probabilities

Modern Transformer Variants

Evolution Beyond the Original

BERT (2018)

Encoder-only
Bidirectional training
Masked language modeling
Classification tasks

GPT (2018)

Decoder-only
Autoregressive generation
Causal language modeling
Text generation

T5 (2019)

Encoder-decoder
Text-to-text framework
All tasks as generation
Unified approach

Vision Transformer (2020)

Encoder-only
Image patches as tokens
Self-attention for vision
Beyond NLP applications

Key Innovations by Model:

BERT: Bidirectional context, pre-training + fine-tuning
GPT: Scaling laws, in-context learning, emergent abilities
T5: Text-to-text unified framework
ViT: Transformers for computer vision

Modern Trend: Larger models (GPT-3: 175B parameters, GPT-4: ~1.7T parameters)

Training and Optimization

Making Transformers Work at Scale

Pre-training Objectives:

Masked Language Model (MLM): Predict masked tokens (BERT)
Causal Language Model: Predict next token (GPT)
Sequence-to-Sequence: Denoising objectives (T5)
Prefix LM: Bidirectional encoder + autoregressive decoder

Optimization Challenges:

Memory: $O(n^2)$ attention matrix storage
Computation: $O(n^2 \cdot d)$ attention complexity
Gradient Flow: Deep networks need careful initialization
Training Stability: Layer norm, residual connections crucial

Solutions:

Gradient Checkpointing: Trade computation for memory
Mixed Precision: FP16/BF16 training
Efficient Attention: Linear attention, sparse attention patterns
Model Parallelism: Distribute large models across devices

$$\text{Training Cost} \propto \text{Parameters} \times \text{Data} \times \text{Compute}$$

Key Takeaways

Why Transformers Changed Everything

Parallelization:
- Self-attention enables parallel processing of sequences
Long-Range Dependencies:
- Direct connections between any two positions
Scalability:
- Architecture scales well with data and compute
Transfer Learning:
- Pre-train on large corpora, fine-tune for specific tasks
Versatility:
- Works for NLP, vision, multimodal tasks
Emergent Abilities:
- Large models show unexpected capabilities

Impact: Transformers are the foundation of modern AI—from ChatGPT to DALL-E to AlphaFold

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

This simple equation revolutionized artificial intelligence

            Future Directions: Efficient attention mechanisms, multimodal transformers, reasoning capabilities
        