Beyond Value Functions: Single-Loop Bilevel Optimization under Flatness Conditions

Tianyi Chen (Cornell University / RPI) · Liuyuan Jiang (University of Rochester) · Quan Xiao (Rensselaer Polytechnic Institute) · Lisha Chen (University of Rochester)

bilevel optimization computational efficiency convergence prooffine-tuning algorithmsfully first-order penalty-based approachgenerative modelshessian vector calculationlandscape analysislarge language modelnested loop updatesrelaxed flatness conditionrepresentation learningsingle-loop updatesstate-of-the-art methodsvalue function-free algorithm

Bilevel optimization, a hierarchical optimization paradigm, has gained significant attention in a wide range of practical applications, notably in the fine-tuning of generative models. However, due to the nested problem structure, most existing algorithms require either the Hessian vector calculation or the nested loop updates, which are computationally inefficient in large language model (LLM) fine-tuning. In this paper, building upon the fully first-order penalty-based approach, we propose an efficient value function-free (\textsf{PBGD-Free}) algorithm that eliminates the loop of solving the lower-level problem and admits fully single-loop updates. Inspired by the landscape analysis of representation learning-based LLM fine-tuning problem, we propose a relaxed flatness condition for the upper-level function and prove the convergence of the proposed value-function-free algorithm. We test the performance of the proposed algorithm in various applications and demonstrate its superior computational efficiency over the state-of-the-art bilevel methods.