Reproduce “EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning” on OpenResearch

About this paper

Siddharth Aravindan, Dixant Mittal, Wee Sun Lee

The paper introduces Event-based Variational Distributions for Exploration (EVaDE), a set of three Gaussian dropout-based convolutional layers designed to perturb the reward function in model-based reinforcement learning. EVaDE-SimPLe achieved a mean Human-Normalized Score of 0.682 on the 100K Atari test suite, outperforming existing baselines by up to 79% and surpassing human performance in 5 games.

Key insights

• For exploration, variational distributions do not need to be exact posterior approximations but should be designed to generate diverse, high-return trajectories that perturb the agent out of local optima.
• Leveraging general domain knowledge about object-based environments allows for structured exploration mechanisms that target specific interaction types, event importance, and positional variations.
• Gaussian multiplicative dropout can be effectively used to learn and induce specific types of uncertainty in the reward function, enabling principled and efficient exploration.

Summary by alphaXiv.