Exploring Bayesian Optimization

Abstraction: Gaussian-process surrogate with acquisition functions for expensive black-box optimization

Key points:

• Bayesian Optimization uses a Gaussian Process (GP) surrogate model to approximate an expensive black-box function, then sequentially optimizes an acquisition function to pick the next evaluation point
• Four main acquisition functions covered: Probability of Improvement (PI), Expected Improvement (EI), Thompson Sampling, and GP-UCB (Upper Confidence Bound)
• PI and EI have an exploration-exploitation tradeoff parameter ε/ξ; too-high values degrade performance to near-random behavior
• GP-UCB's cumulative regret is bounded (Srinivas et al.); GP sensitivity to kernel choice is a key practical pitfall (e.g., Matern kernel assumes first-order differentiability)
• Hyperparameter tuning example: grid search required ~17 hours vs BO reaching near-optimal accuracy in ~7 iterations (~105 minutes total)
• Published on Distill (2020) by Agnihotri and Batra; DOI 10.23915/distill.00026

Connections: Distill · Bayesian Optimization · Gaussian Processes · Hyperparameter Tuning

Source: https://distill.pub/2020/bayesian-optimization/