Gradient Descent

concepts · 13 notes linked

Related: Neural Networks · Backpropagation · Deep Learning · Pytorch · Automatic Differentiation · Michael Nielsen · Linear Regression · Machine Learning

Notes

• 2.7. Mathematical optimization: finding minima of functions — Scipy optimization guide covering gradient methods and practical strategies
• A Simple Neural Network - With Numpy in Python — Implementing neural network forward pass and backpropagation from scratch with NumPy
• Automatic Differentiation: The most criminally underused tool in the potential machine learning toolbox? — Reverse-mode autodiff computes gradients at same cost as the forward function
• Gradient Descent Models Are Kernel Machines (Deep Learning) — Deep networks trained by gradient descent are kernel machines
• Gradient descent for linear regression using PyTorch — PyTorch tutorial implementing minibatch SGD for least-squares linear regression
• Gradient descent for linear regression¶ — GD, SGD, and minibatch SGD implementations for least-squares linear regression
• How to implement an RNN (2/2) - Tensor data and non-linearities — From-scratch NumPy RNN with tensor input, tanh activations, and RMSProp optimizer
• Machine Learning Video Library — Caltech indexed video segments covering core ML topics
• Neural Networks and Deep Learning — Chapter 2 derivation of the four backpropagation equations
• Neural Networks and Deep Learning — Introductory neural network tutorial using MNIST handwritten digit recognition
• PyTorch Explained: From Automatic Differentiation to Training Custom Neural Networks | Towards Data Science — PyTorch core abstractions from tensors and autograd to transformer encoder
• Risky Giant Steps Can Solve Optimization Problems Faster | Quanta Magazine — Oversized gradient descent steps with cyclical patterns converge nearly 3x faster
• The Matrix Calculus You Need For Deep Learning — Matrix calculus tutorial covering gradients and Jacobians for neural network training