Reproduce “Reinforcement Learning for Flow-Matching Policies with Density Transport” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Boshu Lei, Kostas Daniilidis, Antonio Loquercio
Researchers from the University of Pennsylvania developed Reinforcement Learning with Density Transport (RLDT), a method for fine-tuning flow-matching policies using reinforcement learning. RLDT consistently outperforms state-of-the-art baselines across various continuous control tasks, including challenging sparse-reward robotic manipulation, by reframing policy improvement as a probability transport problem and enabling stable gradient propagation without explicit log-likelihood computation.
Key insights
- Reinforcement learning policy improvement can be effectively framed as a probability transport problem on the action manifold, aligning with flow-matching principles.
- The repulsive forces from the SVGD kernel are crucial for stable training and diverse exploration, particularly in challenging sparse-reward robotic tasks.
- Balancing gradient contributions across all intermediate denoising steps via expected target prediction leads to faster convergence and greater stability than prior methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.