Reproduce “Information Theoretic Model Predictive Q-Learning” on OpenResearch

About this paper

Mohak Bhardwaj, Ankur Handa, Dieter Fox, Byron Boots

This work introduces Information Theoretic Model Predictive Q-Learning (MPQ), an algorithm that unifies model-based planning with model-free learning by formally connecting information-theoretic MPC and entropy-regularized reinforcement learning. MPQ demonstrates improved sample efficiency and robustness to model bias in continuous control tasks, enabling effective control with shorter planning horizons.

Key insights

• Information-theoretic Model Predictive Control (IT-MPC) and entropy-regularized Reinforcement Learning share a deep theoretical connection that allows for their synergistic combination.
• Using a locally optimized planning procedure (MPPI) with a potentially biased model can provide stable and high-quality targets for Q-function learning.
• A learned infinite-horizon Q-function can effectively serve as a terminal cost in a finite-horizon MPC, enabling robust long-term planning despite model inaccuracies and truncated horizons.

Summary by alphaXiv.