Reproduce “IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rohan Chitnis, Yingchen Xu, Bobak Hashemi, Lucas Lehnert, Urun Dogan, Zheqing Zhu, Olivier Delalleau
IQL-TD-MPC presents an offline model-based reinforcement learning algorithm integrated into a hierarchical Manager-Worker framework. This approach dramatically enhances performance on long-horizon, sparse-reward D4RL antmaze tasks, boosting average scores for multiple baselines from near-zero to over 40.
Key insights
- Planning within a learned, temporally abstract model of the environment significantly improves the performance of model-based RL agents on long-horizon sparse-reward tasks in offline settings.
- Adapting online MBRL algorithms for offline use by incorporating robustness mechanisms from Implicit Q-Learning allows them to effectively plan while mitigating issues like out-of-distribution generalization errors.
- Augmenting a Worker's state with high-level 'intent embeddings' from a pre-trained hierarchical Manager can effectively compensate for the Worker's inherent lack of long-term planning capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.