Neural Networks and Deep Learning — Chapter 4

Abstraction: Visual proof that neural networks with hidden layers can approximate any continuous function

Key points:

• Universal approximation theorem: a single-hidden-layer network with sigmoid neurons can approximate any continuous function f: [0,1]^m -> R^n to arbitrary precision epsilon by adding more hidden neurons
• Proof constructs "bump functions" by pairing neurons with large weights (step function approximation at position s = -b/w); combining bump pairs creates towers in higher dimensions
• Discontinuous functions cannot be approximated; this is rarely a practical limitation since continuous approximations usually suffice
• Two caveats: only approximations (not exact), only continuous functions; more hidden neurons improve accuracy
• Deep networks preferred over shallow despite universality because hierarchical structure better learns knowledge hierarchies needed for real-world tasks like image recognition
• Original proofs by Cybenko (1989, sigmoidal superpositions) and Hornik, Stinchcombe & White (1989, Stone-Weierstrass) were technical; this chapter gives a visual, intuitive explanation

Connections: Michael Nielsen · Universal Approximation Theorem · Neural Networks · Deep Learning

Source: http://neuralnetworksanddeeplearning.com/chap4.html