Reproduce “A Survey on Model-based Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, Yang Yu

This comprehensive survey examines Model-Based Reinforcement Learning (MBRL), particularly its integration with Deep Reinforcement Learning, to address the high sample complexity of DRL. It synthesizes advancements in learning and utilizing environment models, highlighting strategies to enhance sample efficiency, mitigate compounding errors, and broaden RL's applicability to costly real-world tasks.

Key insights

  • Compounding error, where value estimation error grows quadratically with the effective horizon due to recursive use of learned one-step models, is a central challenge in MBRL.
  • Techniques like Lipschitz continuity constraints and distribution matching (e.g., via adversarial learning) can mitigate compounding error, theoretically achieving linear error bounds with respect to the horizon.
  • MBRL principles extend effectively to other RL paradigms, enhancing sample efficiency and robustness in offline RL, goal-conditioned RL, multi-agent RL, and meta-RL.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.