Reproduce “Flow-GRPO: Training Flow Matching Models via Online RL” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang and 1 more
Flow-GRPO adapts online GRPO to flow-matching image generators by converting their deterministic ODE sampling into an SDE with matched timestep marginals, then using shorter trajectories for training while retaining the full inference schedule. It raises SD3.5-M’s GenEval score from 0.63 to 0.95 and cuts reported training time by over fourfold with 10 steps, not inference time.
Key insights
- Online policy gradient RL, when properly integrated, can effectively enhance compositional reasoning and text rendering abilities of flow matching models.
- Injecting stochasticity into deterministic flow models via an ODE-to-SDE conversion provides the necessary exploration for online RL without sacrificing the original model's probabilistic properties.
- Informative reward signals for online RL can be derived from lower-quality, few-step generated samples, allowing for drastic improvements in training efficiency for large generative models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.