Reproduce “SOMBRL: Scalable and Optimistic Model-Based RL” on OpenResearch

About this paper

Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Florian Dörfler, Pieter Abbeel, Andreas Krause

Researchers from ETH Zurich and UC Berkeley introduce SOMBRL, a method that integrates theoretically grounded optimistic exploration into scalable model-based reinforcement learning by leveraging epistemic uncertainty as an intrinsic reward. The approach demonstrates sublinear regret bounds across various RL settings and enables sample-efficient learning on high-dimensional visual control tasks and real-world robotics, such as an RC car parking maneuver learned in 20 episodes.

Key insights

• Maximizing a weighted sum of extrinsic reward and model epistemic uncertainty is theoretically equivalent to performing optimistic exploration.
• Rephrasing optimistic planning as an intrinsic reward based on epistemic uncertainty allows for practical and scalable integration into modern deep Model-Based Reinforcement Learning algorithms.
• The approach provides sublinear regret guarantees for nonlinear dynamics across multiple challenging Reinforcement Learning settings, including discounted infinite-horizon and non-episodic average reward.

Summary by alphaXiv.