Precise Asymptotics and Refined Regret of Variance-Aware UCB

Yingying Fan (University of Southern California) · Yuxuan Han (New York University) · Jinchi Lv (University of Southern California) · Xiaocong Xu (University of Southern California) · Zhengyuan Zhou (New York University)

advanced algorithmsarm-pulling ratesasymptotic characterizationcanonical ucbdecision-making processhigh probability regimeinstabilitymatching regret lower boundmulti-armed bandit non-asymptotic bounds optimalityrefined regret boundregret analysisupper confidence bound-variancevariance estimatesvariance-aware algorithms

In this paper, we study the behavior of the Upper Confidence Bound-Variance (UCB-V) algorithm for the Multi-Armed Bandit (MAB) problems, a variant of the canonical Upper Confidence Bound (UCB) algorithm that incorporates variance estimates into its decision-making process. More precisely, we provide an asymptotic characterization of the arm-pulling rates for UCB-V, extending recent results for the canonical UCB in Kalvit and Zeevi (2021) and Khamaru and Zhang (2024). In an interesting contrast to the canonical UCB, our analysis reveals that the behavior of UCB-V can exhibit instability, meaning that the arm-pulling rates may not always be asymptotically deterministic. Besides the asymptotic characterization, we also provide non-asymptotic bounds for the arm-pulling rates in the high probability regime, offering insights into the regret analysis. As an application of this high probability result, we establish that UCB-V can achieve a more refined regret bound, previously unknown even for more complicate and advanced variance-aware online decision-making algorithms. A matching regret lower bound is also established, demonstrating the optimality of our result.