Module 1: Why Structure Beats Statistics
This module frames the central question of the course: why do machine learning models fail when pushed outside their training distribution, and what mathematical structure can prevent this? You will examine the extrapolation problem, contrast statistical fitting with structural constraint, and encounter the thesis that drives the rest of the course. No category theory is assumed — this module is about building intuition for why the formalism is necessary before introducing it.
Learning Objectives
- Describe the extrapolation failure and explain why more data does not solve it.
- Contrast statistical fitting and structural constraint as competing paradigms for generalization.
- Define the manifold hypothesis and explain its implications for model architecture.
- Explain why morphisms, not weights, are the correct primitive for inductive bias.
- Identify the three conceptual layers of the course: algebra, geometry, and physics.
- Articulate the course thesis in your own words and apply it to a familiar ML failure case.
Materials
Key Concepts
- Extrapolation Failure: When a model performs well inside the training distribution but produces incorrect outputs for structurally novel inputs, even when those inputs are nearby in Euclidean distance.
- Inductive Bias: The set of assumptions a learning system uses to generalize from training examples to unseen data; all models have one, explicitly or implicitly.
- Manifold Hypothesis: The assumption that natural high-dimensional data (images, language, molecules) concentrates near a low-dimensional manifold embedded in the ambient space.
- Statistical Fitting: The paradigm of minimizing empirical loss over training data; powerful within distribution, unreliable at the boundary.
- Structural Constraint: The paradigm of restricting the hypothesis space using mathematical invariants so that only physically or logically possible models are reachable.
- Morphism: A structure-preserving map between objects; the central primitive of categorical modeling; encodes what is allowed to change and what must be preserved.
- The Course Thesis: The model is the category; generalization is sheaf extension along morphisms; training is finding the critical fixed point of a coarse-graining functor.
Central Concepts from Prerequisites
- Machine learning: training/test split, overfitting, generalization error, loss functions
- Linear algebra: high-dimensional spaces, basis, projection, curse of dimensionality
- Graduate algorithms: computational complexity, tractability