Reproduce “Residual-MPPI: Online Policy Customization for Continuous Control” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pengcheng Wang, Chenran Li, Catherine Weaver, Kenta Kawamoto, Masayoshi Tomizuka, Chen Tang, Wei Zhan

The paper introduces Residual-MPPI, an algorithm that enables online policy customization for continuous control tasks by integrating Residual Q-learning with Model Predictive Path Integral. It allows adaptation of pre-trained policies to new objectives with minimal data, without requiring knowledge of the original reward function, and was demonstrated on tasks including customizing a champion-level racing AI in Gran Turismo Sport.

Key insights

  • Model Predictive Path Integral (MPPI) can effectively approximate maximum-entropy optimal policies, making it a suitable planning component for RQL's objective.
  • The `log π` term within the augmented reward function serves as an effective residual regularization, preserving the original policy's characteristics and long-term optimality during customization.
  • Online fine-tuning of the dynamics model is crucial for robust performance in few-shot settings, adapting the model to states encountered by the customized policy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.