Reproduce “Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P. Adams, Sergey Levine

This research reveals that generalization in reinforcement learning is difficult because an agent's uncertainty about the environment effectively transforms fully-observed problems into partially observed ones. Researchers from UC Berkeley, Princeton University, and Facebook AI Research formalize this as an "epistemic POMDP" and propose LEEP, an ensemble-based algorithm that improves generalization performance on Procgen benchmarks and prevents overfitting with limited training data.

Key insights

  • An agent's epistemic uncertainty about the underlying environment parameters implicitly introduces partial observability, making even fully-observed contextual MDPs behave like POMDPs during generalization.
  • Bayes-optimal policies for generalization are generally non-Markovian and potentially stochastic, contrasting with the typically deterministic and Markovian optimal policies in standard MDPs.
  • Policies optimized for individual MDPs, as in conventional RL, can be arbitrarily sub-optimal for generalization because they fail to account for the need for information-gathering actions in an uncertain environment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.