Neural Networks and Deep Learning

Abstraction: Chapter 2 derivation of the four backpropagation equations

Key points:

• Backpropagation was introduced in the 1970s but gained prominence via the 1986 Rumelhart, Hinton & Williams paper in Nature
• Four fundamental equations (BP1-BP4) use the chain rule: output error delta^L, layer-to-layer error propagation, gradient w.r.t. bias equals delta, gradient w.r.t. weight = a_in * delta_out
• Key insight: one forward pass + one backward pass computes all partial derivatives simultaneously, vs. O(n_weights) forward passes for finite-difference approximation
• The Hadamard (elementwise) product is central to the matrix form of backprop
• Saturated neurons (sigmoid output near 0 or 1) cause near-zero sigma-prime, slowing weight learning — a key motivation for non-sigmoid activations
• Includes Python/NumPy implementation of update_mini_batch and backprop methods using stochastic gradient descent with mini-batches

Connections: Michael Nielsen · Geoffrey Hinton · Backpropagation · Neural Networks · Gradient Descent

Source: http://neuralnetworksanddeeplearning.com/chap2.html