Reproduce “Accelerated Policy Learning with Parallel Differentiable Simulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, Miles Macklin

A new policy learning algorithm, Short-Horizon Actor-Critic (SHAC), leverages a high-performance parallel differentiable simulator to accelerate deep reinforcement learning for complex control tasks. It achieves notable improvements in both sample and wall-clock time efficiency, demonstrating a 17x speedup and 382x sample efficiency gain over PPO for muscle-actuated humanoids.

Key insights

  • Differentiable simulation can effectively accelerate policy learning in continuous, contact-rich control tasks when integrated with an actor-critic framework that intelligently manages gradient propagation.
  • Truncating gradient backpropagation paths and using a smooth value function are crucial for overcoming issues like exploding/vanishing gradients, local minima, and discontinuities inherent in complex physical dynamics.
  • High-performance parallel differentiable simulators are essential to translate the theoretical sample efficiency benefits of analytical gradients into practical wall-clock time reductions for real-world applications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.