Reproduce “Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Franki Nguimatsia Tiofack, Théotime Le Hellard, Fabian Schramm, Nicolas Perrin-Gilbert, Justin Carpentier

Researchers from Inria, DI-ENS, and Sorbonne Universit developed Guided Flow Policy (GFP), an offline reinforcement learning algorithm that achieves state-of-the-art performance by using value-aware behavior cloning to selectively learn from high-value actions in datasets. It demonstrated substantial gains on tasks with suboptimal and noisy data across OGBench, D4RL, and Minari benchmarks.

Key insights

  • Incorporating value-awareness directly into behavior regularization allows for selective learning from beneficial data trajectories, which is crucial for improving performance on suboptimal or noisy offline datasets.
  • A bidirectional guidance mechanism, where the generative VaBC policy regularizes the actor and the actor/critic refines VaBC's understanding of high-value actions, enables more stable and effective learning.
  • Expressive generative policies can be effectively integrated into offline RL by distilling them into efficient one-step actors, mitigating high computational costs without sacrificing the capacity to capture complex action distributions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.