Flow-GRPO: Training Flow Matching Models via Online RL

Wanli Ouyang (Shanghai AI Lab) · Jie Liu (City University of Hong Kong) · Pengfei Wan (Kuaishou Technology) · Di ZHANG (Kuaishou Technology) · Xintao Wang (Applied Research Center, Tencent PCG) · Jiaheng Liu (Nanjing University) · Gongye Liu (The Hong Kong University of Science and Technology) · Jiajun Liang (Kuaishou) · Yangguang Li (Shanghai AI Laboratory)

compositional generationdenoising reduction strategyflow matching modelsflow-grpogeneval accuracyhuman preference alignmentimage qualityode-to-sde conversiononline policy gradientreinforcement learning reward hackingrl-tuned sd3.5-msampling efficiencystatistical samplingstochastic differential equationvisual text rendering

We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from $63\%$ to $95\%$. In visual text rendering, accuracy improves from $59\%$ to $92\%$, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.