Multi-Objective One-Shot Pruning for Large Language Models

Weiyu Chen (The Hong Kong University of Science and Technology) · James Kwok (Hong Kong University of Science and Technology) · Zhenguo Li (Huawei Noah's Ark Lab, Hong Kong) · Hansi Yang (Department of Computer Science and Engineering, Hong Kong University of Science and Technology) · Yunhao Gou (Hong Kong University of Science and Technology) · Han Shi (Hong Kong University of Science and Technology) · Enliang Hu (Yunnan Normal University)

baseline methodscapability trade-offscomputational resourcescore supportmulti-faceted applicationsmulti-objective optimizationone-shot pruningoptimization problem.pareto setpruned modelsresource-constrained environmentssparsity levelsspecialized supportsuperior performance

Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but require substantial computational resources, limiting their deployment in resource-constrained environments. While one-shot pruning methods can reduce model size without expensive retraining, they typically optimize for single objectives, ignoring LLMs' multi-faceted applications. We introduce Multi-Objective One-Shot Pruning (MOSP), which formulates LLM pruning as a multi-objective optimization problem. MOSP efficiently generates a Pareto set of pruned models representing different capability trade-offs, allowing users to select solutions aligned with their preferences. The proposed approach identifies share core support while enabling specialized support. Experiments across various LLMs and sparsity levels demonstrate MOSP's superior performance in navigating multi-objective trade-offs compared to baseline methods.