Reproduce “ReFPO: Reflow Regularization for Flow Matching Policy Gradients” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ge Wang, Yibo Peng, Fan Feng, Shenhao Yan, Chengsi Yao, Jiahao Yang, Honghao Cai, Yiming Zhao and 5 more
ReFPO enhances Flow Matching Policy Gradients (FPO) by incorporating an explicit geometric regularizer, improving the stability of online reinforcement learning and enabling high-fidelity one-step action inference. The approach maintains performance comparable to multi-step methods while significantly reducing latency and mitigating training instabilities across continuous control benchmarks.
Key insights
- The authors identified that FPO inherently functions as an "implicit advantage-weighted Reflow process," where policy updates encourage straightening, but without explicit, consistent control over the overall flow geometry.
- Explicitly regularizing the flow field to be straighter, independent of advantage weighting, directly reduces the local truncation error of one-step ODE solvers, enabling high-fidelity single-step action generation.
- Consistently promoting straighter velocity fields stabilizes the PPO-style inner optimization loop by reducing volatility in the CFM-based proxy ratio, mitigating training instabilities and proxy-ratio spikes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.