Sparse Models, The Math, And A New Theory For GroundBreaking AI

Abstraction: Poggio's theory that compositional sparsity explains deep network effectiveness

Key points:

• MIT's Tomaso Poggio (Eugene McDermott Professor, McGovern Institute, CSAIL) argues a theory-first approach is needed for "deep explainability" in AI and ML systems
• His conjecture: every practically computable function (Turing-computable in non-exponential time) is compositionally sparse and can be approximated by a deep network with matching sparse connectivity — avoiding the curse of dimensionality
• Compositionally sparse functions decompose into binary-tree hierarchies of low-dimensional subsets of variables; dense networks without such structure suffer exponential scaling
• Convolutional networks are effective precisely because their architecture mirrors compositional sparsity in image data; dense networks are comparatively poor
• Transformers may exploit the same principle via self-attention: at each layer, attention selects a sparse subset of tokens, implicitly discovering the sparse compositional structure
• Poggio advocates for principled theory to reveal shared intelligence principles between large language models and human cognition

Connections: Mit · Tomaso Poggio · Curse Of Dimensionality · Sparse Models · Deep Learning · Explainability

Source: https://www.forbes.com/sites/johnwerner/2023/08/17/sparse-models-the-math-and-a-new-theory-for-ground-breaking-ai/