Reproduce “Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Arthur Guez, David Silver, Peter Dayan

BAMCP approximates Bayes-adaptive reinforcement-learning planning by Monte Carlo tree search: each simulation samples one coherent environment model, while lazy sampling instantiates only the transition details it encounters. This avoids repeated belief updates and full model construction. On Dearden’s Maze, it reports 965.2 ± 73 versus BFS3’s 240.9 ± 46, with 1.5 seconds per step; this is not finite-budget Bayes-optimality.

Key insights

  • Filtering samples at the root of the search tree through root sampling effectively bypasses the need for explicit, expensive posterior belief updates within every node of the Monte-Carlo tree.
  • Sampling model parameters on-demand (lazy sampling) significantly improves computational efficiency, particularly in models with many parameters where only a subset is relevant for any given trajectory.
  • Combining Monte-Carlo tree search with these sampling efficiencies allows for practical Bayes-optimal planning with theoretical convergence guarantees, even in complex domains where traditional Bayesian RL methods are intractable.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.