Reproduce “Curiosity-Driven Exploration via Latent Bayesian Surprise” on OpenResearch

About this paper

Pietro Mazzaglia, Ozan Catal, Tim Verbelen, Bart Dhoedt

Researchers from IDLab, Ghent University, developed Latent Bayesian Surprise (LBS), an intrinsic reward mechanism for reinforcement learning that quantifies information gain in a learned latent space. This approach addresses the computational inefficiency of previous Bayesian surprise methods and the "NoisyTV problem" of prediction-error-based curiosity, leading to improved exploration in both continuous control tasks and high-dimensional video games, and robustness in stochastic environments.

Key insights

• Computing Bayesian surprise in a latent space significantly reduces computational costs compared to parameter-space methods, making it more practical for complex RL problems.
• Deriving intrinsic reward from the information gain about latent dynamics (KL divergence) inherently provides robustness against uninformative stochasticity, effectively mitigating the "NoisyTV problem."
• An agent can be effectively driven to explore by incentivizing transitions that yield the most information about its internal model of environmental dynamics.

Summary by alphaXiv.