Reproduce “MODIP: Efficient Model-Based Optimization for Diffusion Policies” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zakariae El Asri, Philippe Gratias-Quiquandon, Nicolas Thome, Olivier Sigaud

MODIP (Model-Based Optimization for Diffusion Policies) is a framework that fine-tunes Diffusion Policies (DPs) by leveraging model-based planning and distilling improved trajectories back into the DP. This approach enhances DPs beyond imitation learning while addressing computational inefficiencies in online settings, achieving competitive or superior performance across D4RL MuJoCo, Kitchen, and RoboMimic tasks, and speeding up inference by 2.90x and training by 1.6x on HalfCheetah.

Key insights

  • Indirect policy improvement through MPC-to-policy distillation and behavioral cloning offers a stable and effective mechanism for enhancing Diffusion Policies beyond their initial imitation learning capabilities.
  • Replacing policy-dependent Q-value estimates with a terminal state-value function V(s) in MPC significantly boosts planning efficiency and improves final task performance.
  • Decoupling critic learning from the Diffusion Policy, by forming TD targets using state-value estimates, dramatically reduces overall training time by avoiding expensive multi-step DP queries.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.