Module 12: Statistical Mechanics and the Renormalization Group
This module introduces the statistical mechanics required to cash the physics metaphors of the course precisely. The Ising model provides a concrete playground for partition functions, phase transitions, and criticality. The Renormalization Group is then developed as a coarse-graining functor whose fixed points define universality classes. The module culminates in a precise mapping between training dynamics and phase transitions, and an interpretation of well-trained weight matrices using random matrix theory — giving a measurable signature of when a network has reached the critical fixed point.
Learning Objectives
- Define the Ising model Hamiltonian and compute the partition function Z for small systems.
- Describe a second-order phase transition and identify the critical temperature T_c and critical exponents.
- Explain the Renormalization Group as a coarse-graining transformation and identify it as a functor.
- Identify the fixed points of the RG flow and connect them to universality classes.
- Map the three phases of neural network training (disordered, critical, ordered) to phases of a statistical mechanical system.
- Interpret the spectral distribution of weight matrices using Marchenko-Pastur and explain what deviation from it signals.
Materials
Key Concepts
- Ising Model: A lattice model of binary spins σ_i ∈ {±1} with Hamiltonian H = −J Σ σ_i σ_j − h Σ σ_i; the canonical model for studying phase transitions.
- Partition Function Z: Z = Σ_σ exp(−βH(σ)); the sum over all spin configurations weighted by the Boltzmann factor; all thermodynamic quantities derive from log Z.
- Phase Transition: A qualitative change in macroscopic behavior at a critical temperature T_c; the order parameter (magnetization) is zero above T_c and nonzero below it.
- Renormalization Group: A coarse-graining transformation that integrates out short-scale degrees of freedom and rescales; the RG flow is a functor on the space of theories.
- Fixed Point: A theory invariant under the RG transformation; defines a universality class; all systems that flow to the same fixed point share the same critical exponents.
- Universality Class: The set of all physical systems sharing the same RG fixed point and therefore the same large-scale critical behavior, regardless of microscopic differences.
- Marchenko-Pastur Distribution: The limiting spectral distribution of random rectangular matrices; weight matrices of well-trained networks that have reached the critical fixed point follow this distribution.
Central Concepts from Prerequisites
- Modules 8–9: Kan extensions, coends, partition functions
- Probability: distributions, expectations, large deviations
- Linear algebra: eigenvalue spectra, random matrix theory
- No prior physics assumed; all definitions provided