Automatic Differentiation: The most criminally underused tool in the potential machine learning toolbox?

Abstraction: Reverse-mode autodiff computes gradients at same cost as the forward function

Key points:

• Reverse-mode autodiff produces an exact gradient subroutine with the same computational complexity as the original function, without finite-difference approximation or symbolic expansion
• Backpropagation for neural networks is a special case of autodiff; autodiff predates backprop
• Works by applying the calculus chain rule to the expression graph of elementary operations in a subroutine
• ML papers that manually derive gradients restrict the field to architectures humans can differentiate by hand, excluding harder models like image boundary algorithms
• Distinct from symbolic differentiation: autodiff can differentiate algorithms (loops, branches), producing no expression blowup; C++ users can use David Gay's RAD toolbox

Connections: Automatic Differentiation · Backpropagation · Gradient Descent · Machine Learning

Source: http://justindomke.wordpress.com/2009/02/17/automatic-differentiation-the-most-criminally-underused-tool-in-the-potential-machine-learning-toolbox/