Reproduce “Training Efficient Controllers via Analytic Policy Gradient” on OpenResearch

About this paper

Nina Wiedemann, Valentin Wüest, Antonio Loquercio, Matthias Müller, Dario Floreano, Davide Scaramuzza

This research introduces a method for training control policies using Analytic Policy Gradients (APG) that achieves trajectory tracking performance similar to Model Predictive Control (MPC) with significantly faster inference times. The approach stabilizes APG training through a concurrent receding-horizon strategy and curriculum learning, outperforming Reinforcement Learning (RL) in accuracy, efficiency, and sample complexity across various robotic tasks.

Key insights

• The concurrent receding-horizon training strategy provides superior stability and performance compared to traditional autoregressive APG approaches, especially for longer prediction horizons.
• Curriculum learning is crucial for stabilizing APG training, preventing convergence to local minima and enabling the controller to learn robustly across varying initial conditions and task complexities.
• APG exhibits orders of magnitude higher sample efficiency than both model-free and model-based RL, making it highly effective for learning directly from high-dimensional inputs and for rapid adaptation to new system dynamics.

Summary by alphaXiv.