Reproduce “First Order Model-Based RL through Decoupled Backpropagation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Joseph Amigo, Rooholla Khorrambakht, Elliot Chane-Sane, Nicolas Mansard, Ludovic Righetti

DMO introduces a first-order gradient reinforcement learning method that decouples trajectory generation using a high-fidelity simulator from gradient computation via a learned differentiable model, effectively mitigating compounding prediction errors. This approach achieves over tenfold higher sample efficiency and up to a 20% reduction in wall-clock training time compared to model-free baselines, demonstrating robust sim-to-real transfer on a quadrupedal robot.

Key insights

  • Decoupling trajectory generation from gradient computation using different models significantly mitigates compounding prediction errors, improving learning stability and performance.
  • Leveraging the realism of high-fidelity simulators for forward passes and the differentiability of learned models for backward passes combines the strengths of both approaches.
  • This method provides a practical way to achieve analytical policy gradients without the need for a fully differentiable high-fidelity simulator, making FoG-RL more accessible.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.