Module 1 Glossary: Why Structure Beats Statistics

Key terms introduced in this module. Terms from later modules that are previewed here will be defined more precisely when they appear.

Extrapolation Failure: The failure mode in which a trained model produces incorrect or meaningless outputs for inputs that are outside its training distribution. Distinct from interpolation error: the model may perform well on all training and held-out test points yet fail catastrophically on structurally novel inputs. More data does not fix extrapolation failure; the model's hypothesis space must be restricted to exclude structurally wrong generalizations.
Distribution Shift: A change in the statistical distribution of inputs between training time and deployment time. Covariate shift (the input distribution changes but the conditional output distribution does not), label shift, and concept drift are subtypes. Extrapolation failure is a severe form of distribution shift where the test inputs fall outside the support of the training distribution entirely.
Inductive Bias: The set of assumptions a learning algorithm uses to generalize from the training examples it has seen to unseen examples. Every learning algorithm has an inductive bias — there is no assumption-free learner (No Free Lunch Theorem). The central claim of this course is that the correct inductive bias for physical and structured domains is not learned from data but encoded as the morphism structure of a category.
Hypothesis Space: The set of all functions the learning algorithm can in principle output. The hypothesis space is determined by the model architecture before any training. Topological invariants (Module 10) and categorical morphisms (Module 2 onward) constrain the hypothesis space by eliminating entire connected components of structurally impossible models.
Manifold Hypothesis: The empirical observation and theoretical assumption that natural high-dimensional data — images, language, molecular structures — does not fill the ambient space uniformly but concentrates near a low-dimensional manifold M ↪ ℝ^d where dim(M) ≪ d. If the manifold hypothesis holds, the effective complexity of the learning problem is determined by dim(M), not d. This is the bridge between high-dimensional observations and tractable structural modeling.
Manifold: A topological space that locally resembles Euclidean space. Formally, an n-dimensional manifold is a topological space in which every point has a neighborhood homeomorphic to ℝ^n. The manifold hypothesis asserts that natural data lives on such a space. The geometry and topology of this manifold encode the structural constraints of the domain.
Statistical Fitting: The paradigm of machine learning in which a model is defined by minimizing an empirical loss function over training data. The model has no representation of the structural laws of the domain — it approximates a target function from examples alone. Statistical fitting is powerful within the training distribution and fragile outside it. EBMs, neural networks, and kernel methods all operate in this paradigm by default.
Structural Constraint: A restriction on the hypothesis space derived from mathematical invariants of the domain rather than from data. Examples: requiring a model to be G-equivariant for a symmetry group G (Module 3); requiring predicted defect types to belong to π₁ of the order parameter space (Module 10); requiring locally consistent model updates to extend globally (Module 11). Structural constraints make the model correct by construction rather than approximately correct by learning.
Morphism: A structure-preserving map between objects in a category. The central primitive of categorical modeling. Morphisms encode what is allowed to change between two objects (and what must be preserved). In a category of physical states, morphisms are valid physical transformations. Specifying the morphisms IS specifying the model — the Kan extension (Module 8) then determines all generalization from training data to unseen states.
Invariant: A quantity or property preserved by all morphisms in a category. Topological invariants (Module 10) are preserved by continuous deformations. Algebraic invariants are preserved by structure-preserving maps. The key use of invariants in this course: if a model violates a known invariant of the domain, it is provably wrong — no training procedure can compensate for a structurally incorrect hypothesis space.
The Course Thesis: The central claim stated in three equivalent formulations: (1) The model is the category — specifying objects and morphisms is specifying the model. (2) Generalization is sheaf extension along morphisms — valid extrapolation is the unique consistent extension of known values along the morphism structure. (3) Training is finding the critical fixed point of a coarse-graining functor — learning is the process of converging to the RG fixed point that represents the domain's structural invariants.