Reproduce “EMI: Exploration with Mutual Information” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, Hyun Oh Song

EMI introduces an intrinsic-motivation method for sparse-reward reinforcement learning that learns compact state and action embeddings through mutual-information objectives. By imposing approximately linear transition dynamics in the embedding space and rewarding residual prediction error, it improves exploration across continuous-control and visual Atari tasks, including a mean score of 387 on Montezuma’s Revenge and 218.1 on SparseHalfCheetah.

Key insights

  • Mutual-information learning prevents the latent representation from collapsing and focuses it on transition-relevant structure rather than generic visual similarity.
  • The error model is important for avoiding repeated intrinsic rewards from stochastic or structurally mismatched transitions; removing it substantially reduces performance and destabilizes learning.
  • A low-dimensional latent space can organize semantically related observations: experiments show room separation in Montezuma’s Revenge and smooth state trajectories in Frostbite.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.