Reproduce “Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruijie Hao, Longfei Zhang, Yang Dai, Yang Ma, Xingxing Liang, Guangquan Cheng

A new reinforcement learning framework called FP-DRL combines flow-based policies with a quantile-based distributional critic, enabling agents to learn complex, multimodal action distributions guided by the full spectrum of returns. This framework consistently achieves state-of-the-art performance on several MuJoCo continuous control benchmarks, demonstrating improved stability and efficiency.

Key insights

  • Flow-based models can serve as computationally efficient and expressive policy representations for capturing multimodal action distributions in online reinforcement learning.
  • Utilizing a distributional critic, specifically a quantile-based approach, offers a significantly richer learning signal than mean returns, crucial for effectively guiding complex policies.
  • Combining flow-based policies with distributional reinforcement learning yields a synergistic framework that enhances both learning stability and final performance in continuous control tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.