The Singular Value Decompositions of Transformer Weight Matrices

Abstraction: SVD of GPT-2 weight matrices reveals interpretable semantic directions

Key points:

• Taking the SVD of OV circuit and MLP weight matrices in GPT-2, then projecting singular vectors to token embedding space, yields highly interpretable semantic clusters for ~70-80% of top singular directions in later layers
• OV circuit heads tend to specialize in a broad semantic concept with each singular vector encoding a subdirection; MLP blocks are more polysemantic but semantically deeper (meaningful up to ~100 singular directions vs ~50 for OV)
• Both positive and negative singular vectors encode interpretable — sometimes antipodal — semantic directions (e.g., fire vs. ice in head 3 layer 22)
• A natural-language query tool can locate weight directions corresponding to a given concept via cosine similarity in embedding space
• Singular vectors can be deleted via rank-one updates (M_hat = M - S_i U_i V_i^T) with targeted effects on related logits, with minimal collateral semantic impact
• GPT-3 was used for automated scalable labelling of SVD directions as proof of concept; ~70-80% interpretability rate held across GPT-2 small, medium, and large

Connections: GPT-2 · Openai · Mechanistic Interpretability · Transformers · Superposition

Source: https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight