Reproduce “HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zengjue Chen, Peidong Liu, Jiawei Li, Qi Wang

HaWMPO is a reinforcement learning pipeline that enhances generalist robot policies by explicitly addressing prediction hallucinations in world models, leading to more robust policy optimization. It quantifies the reliability of imagined trajectories and modulates reward signals accordingly, achieving higher success rates on manipulation tasks in both simulation and real-world experiments.

Key insights

  • Explicitly quantifying the reliability of world model-generated imagined rollouts provides a crucial signal for improving policy learning in simulated environments.
  • Modulating reward signals based on hallucination levels prevents policies from exploiting world model biases, leading to more stable and effective reinforcement learning.
  • A dedicated hallucination detection model (HAM) can provide a reliable, reference-free assessment of video chunk quality, enabling its use in an online RL loop.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.