Transformer Explainer: LLM Transformer Model Visually Explained

transformersgpt-2self-attentionneural-networksvisualizationeducation

Abstraction: Interactive visual walkthrough of Transformer/GPT-2 architecture

Key points:

Interactive tool running a live GPT-2 (small) model — 124M params, 12 Transformer blocks, 12 attention heads, 768-dim embeddings, 50,257-token vocabulary — in-browser via ONNX Runtime, built from Andrej Karpathy's nanoGPT; created at Georgia Tech (Aeree Cho et al.).
Three key components: Embedding (tokenize → 768-dim token embeddings from a ~39M-param matrix + positional encoding), Transformer Blocks (multi-head self-attention + MLP), and Output Probabilities (linear + softmax over vocab).
Self-attention uses Query/Key/Value (search-engine analogy); Q·K dot product gives attention scores, scaled and masked (upper triangle → −∞ to block future tokens), softmax + dropout, then multiplied by V and concatenated across heads.
MLP expands 768→3072 (4x) then back to 768 with GELU activation, processing each token independently; auxiliary features: layer normalization (twice per block), dropout, residual connections (from ResNet, 2015).
Sampling controlled by temperature (>1 = more creative/random, <1 = deterministic), plus top-k and top-p sampling; transformer from "Attention is All You Need" (2017), core to GPT, Llama, Gemini.

Connections: GPT-2 · Andrej Karpathy · Georgia Tech · Transformers · Self Attention · Large Language Models · Tokenization