Neural Networks and Deep Learning
neural-networksdeep-learninggradient-descentmnistperceptronsigmoid
Abstraction: Introductory neural network tutorial using MNIST handwritten digit recognition
Key points:
- Perceptrons (developed by Frank Rosenblatt in 1950s–60s) take binary inputs with weights and a threshold/bias to produce binary output; networks of perceptrons can compute any logical function (NAND-universal)
- Sigmoid neurons smooth perceptron step function to σ(z) = 1/(1+e^−z), allowing small weight/bias changes to produce small output changes — enabling gradient-based learning
- The quadratic cost function C(w,b) = (1/2n)Σ‖y(x)−a‖² (MSE) is minimized via gradient descent: update rule v → v − η∇C
- Stochastic gradient descent (SGD) estimates ∇C from random mini-batches rather than all training data, giving ~6,000x speedup on MNIST (n=60,000, m=10)
- Three-layer network with 784 inputs (28×28 pixels), hidden layer of ~15 neurons, and 10 output neurons achieves >96% accuracy on MNIST in 74 lines of Python/NumPy
- MNIST dataset: 60,000 training images + 10,000 test images of handwritten digits, assembled from NIST data by Yann LeCun et al.
Connections: Michael Nielsen · Mnist · Neural Networks · Gradient Descent · Stochastic Gradient Descent · Sigmoid Function