Reproduce “Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Benjamin Yu, Jackie Liu, Justin Cui
Smart-GRPO optimizes noise sampling for reinforcement learning (RL) on flow-matching generative models by iteratively refining a noise distribution to favor high-reward generations. This method improves the efficiency and stability of RL fine-tuning, leading to higher-quality image generation aligned with human preferences compared to prior random noise approaches.
Key insights
- Optimizing noise perturbations for flow-matching models significantly enhances the efficiency and stability of reinforcement learning by reducing wasted training signal from low-quality generations.
- Iterative refinement of the noise distribution allows for a progressive concentration of sampling probability around more informative (higher-reward) noise regions.
- A one-step approximation for reward calculation provides an efficient way to estimate image quality during noise selection, particularly effective at later timesteps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.