Beyond automatic differentiation

autoboundoptimizationautomatic-differentiationjaxgoogle-research

Abstraction: Google's AutoBound computes polynomial function bounds enabling hyperparameter-free optimizers

Key points:

AutoBound algorithm computes polynomial upper and lower bounds on arbitrary functions over a user-specified interval, generalizing Taylor mode automatic differentiation
Derives optimal polynomial bounds (tightest possible given the Taylor series constraints) for elementary functions including exp, log, ReLU, Swish using a novel interval-polynomial chain rule
Applications include: (1) SafeRate meta-optimizer that wraps Adam with a guaranteed monotonically-decreasing learning rate at ~2x wall-time cost; (2) verified numerical integration; (3) sharper proofs of Jensen's inequality
SafeRate requires only one additional forward pass per step; no hyperparameter tuning needed
Implemented as open-source JAX library at github.com/google/autobound
Paper: "Automatically Bounding The Taylor Remainder Series: Tighter Bounds and New Applications" (arXiv:2212.11429)

Connections: Google Research · Automatic Differentiation · Optimization · Machine Learning Theory