Reproduce “RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang, Sha Zhang, Jiajun Deng, Yanyong Zhang

RoMAN-Flow introduces a framework that enables efficient and practical application of Autoregressive Normalizing Flow policies for offline robotic manipulation. It integrates a sampling-free offline reinforcement learning algorithm for policy refinement and a one-step distillation method to significantly accelerate inference, achieving strong performance on both simulated and real-world tasks with an 8.55x speedup in action generation.

Key insights

  • Autoregressive Normalizing Flows (AR-NFs) can serve as effective, continuous-action policy representations for robotic manipulation if their inherent efficiency challenges are systematically addressed.
  • Offline reinforcement learning can effectively refine AR-NF policies through a sampling-free, advantage-weighted likelihood optimization paradigm, bypassing the need for costly sequential policy rollouts during training.
  • The high inference latency of autoregressive generation can be significantly reduced by distilling the trained policy into a parallel, one-step student model, preserving or even enhancing performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.