2.7. Mathematical optimization: finding minima of functions

Abstraction: Scipy optimization guide covering gradient methods and practical strategies

Key points:

• Convex functions guarantee any local minimum is global; non-convex problems are much harder. Smooth functions easier than non-smooth; ill-conditioned functions harder.
• Gradient descent oscillates across valleys; conjugate gradient adds friction (uses last two gradient values) to reduce sharp turns — use method="CG" in scipy.optimize.minimize.
• Newton methods use the Hessian for quadratic approximation and faster convergence but Hessian inversion is costly/unstable above ~250 dimensions; L-BFGS stores a low-rank Hessian approximation instead.
• Nelder-Mead (simplex) is gradient-free, robust to noise, but slower than gradient methods on smooth functions.
• Least-squares problems use scipy.optimize.leastsq (Levenberg-Marquardt); curve_fit wraps this for non-linear regression.
• Practical tips: provide analytic gradient/Hessian when possible (27 vs 108 function evals in example), prescale variables for better conditioning, use check_grad to validate gradients.

Connections: Scipy · Numpy · Mathematical Optimization · Gradient Descent · Convex Optimization · Numerical Methods

Source: http://scipy-lectures.github.io/advanced/mathematical_optimization/