Reproduce “Flow Actor-Critic for Offline Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jongseong Chae, Jongeui Park, Yongjae Shin, Gyeongmin Kim, Seungyul Han, Youngchul Sung

KAIST researchers developed Flow Actor-Critic (FAC), an offline reinforcement learning framework that utilizes flow models for both policy generation and a novel critic penalization based on precise behavior density estimation. This approach effectively mitigates out-of-distribution value overestimation, achieving state-of-the-art results on OGBench and strong performance on D4RL benchmarks, notably improving performance by 238% on "puzzle-3x3-play".

Key insights

  • Normalizing flow models provide superior, high-fidelity density estimates for complex, multi-modal action distributions, enabling robust distinction between in-distribution and OOD actions.
  • Directly penalizing Q-values based on flow-derived behavior density effectively mitigates OOD overestimation, preserving unbiased Q-values in high-confidence regions and conservatively underestimating them in low-confidence areas.
  • The synergistic combination of an expressive one-step flow actor and a conservatively regularized flow critic guides policy optimization towards valuable actions within the data support, enhancing stability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.