CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

Rongrong Ji (Xiamen University, China) · Yuan Xie (East Normal China University) · Zhihang Lin (Xiamen University) · Mingbao Lin (Xiamen University)

completion pruning policy optimizationdynamic completion allocationgpu utilizationgradient calculationgroup relative policy optimization gsm8k math model accuracy policy training reasoning modelsrelative advantagesampling completionsspeedup training costs training efficiency

This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training