Transformers Explained Visually (Part 3): Multi-head Attention, deep dive

Abstraction: Deep dive into multi-head attention mechanics and data dimensions

Key points:

• Attention is used in three places: encoder self-attention, decoder self-attention, and encoder-decoder attention
• Multiple heads achieved by logical (not physical) splitting of Q/K/V matrices; all heads share one Linear layer
• Query Size = Embedding Size / Number of Heads; enables single matrix operation for all heads simultaneously
• Attention Score computation: Q×K matrix multiply → mask → scale by √(Query Size) → softmax → multiply by V
• After per-head computation, head outputs are concatenated by reshaping: (Batch, Head, Seq, Query Size) → (Batch, Seq, Embedding Size)
• Separate head sections of the embedding learn different aspects of word meaning (e.g., gender, cardinality), enabling richer representations

Connections: Transformers · Attention Mechanism · Large Language Models

Source: https://towardsdatascience.com/transformers-explained-visually-part-3-multi-head-attention-deep-dive-1c1ff1024853