Reproduce “Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiayu Chen, Le Xu, Aravind Venugopal, Jeff Schneider

Researchers from The University of Hong Kong, Tsinghua University, and Carnegie Mellon University developed ROMBRL, an offline model-based reinforcement learning algorithm that unifies world model and policy optimization using constrained maximin Stackelberg dynamics. This approach yields policies that are robust to environmental noise, achieving an average normalized score of 77.7 on D4RL MuJoCo tasks with 5% measurement noise and a minimal performance drop of -0.6% from noiseless conditions.

Key insights

  • Unifying world model and policy optimization through constrained maximin Stackelberg dynamics effectively addresses the objective mismatch and enhances policy robustness against environmental perturbations.
  • Introducing a KL-divergence constraint on the adversarial world model prevents it from diverging unrealistically, maintaining its relevance to the offline data and enabling effective learning without real environment interaction.
  • Complex second-order Stackelberg dynamics can be made computationally feasible for practical offline RL through approximations (e.g., Fisher Information) and efficient matrix inversion techniques (e.g., Woodbury identity).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.