Reproduce “Safe Exploration Using Bayesian World Models and Log-Barrier Optimization” on OpenResearch

About this paper

Yarden As, Bhavya Sukhija, Andreas Krause

Cost-Efficient Reinforcement Learning (CERL) introduces a method that integrates Bayesian world models with log-barrier optimization to enable consistently safe exploration in online reinforcement learning. The approach maintains safety throughout the entire learning process and achieves competitive rewards in high-dimensional, partially-observable environments, outperforming existing baselines in ensuring safety during exploration.

Key insights

• Quantifying epistemic uncertainty in the world model and performing pessimistic planning against this uncertainty is crucial for maintaining safety throughout the entire learning trajectory.
• Interior-point constrained optimization techniques, like LBSGD, can empirically ensure that policy iterates remain feasible (safe) during training, even when operating in complex, high-dimensional observation spaces.
• Integrating robust, uncertainty-aware environment models with principled constrained optimization enables sample-efficient and consistently safe reinforcement learning from high-dimensional, partially-observable inputs.

Summary by alphaXiv.