Reproduce “NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning” on OpenResearch

About this paper

Diyuan Shi, Yiqi Tang, Zifeng Zhuang, Donglin Wang

Researchers from Zhejiang University and Westlake University developed NFPO, a method that integrates Normalizing Flows into the on-policy PPO framework for robotic policy learning. NFPO achieves stable multi-modal policy learning by normalizing internal scale network outputs, demonstrates competitive or superior performance against Gaussian PPO in high-dimensional tasks, and successfully transfers to real-world humanoid robots for stable control.

Key insights

• Naive integration of Normalizing Flows into online Reinforcement Learning causes significant training instability due to issues like overfitting to high-advantage samples, exponentially growing internal values, and unbounded network outputs.
• Constraining the output of the scale network within RealNVP via normalization (e.g., `tanh` or `clip`) is crucial for achieving stable and robust Normalizing Flow-based policy optimization.
• Unlike standard PPO, an explicit entropy regularization term and adding action noise during training were found to be unnecessary or even detrimental to NFPO's performance and stability across various robotic tasks.

Summary by alphaXiv.