Reproduce “Better Optimism By Bayes: Adaptive Planning with Rich Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arthur Guez, David Silver, Peter Dayan
The paper compares BAMCP with Thompson Sampling in constructed sequential decision tasks based on mushroom data and a Chinese Restaurant Process mixture. BAMCP uses sampled forward search over possible futures, while Thompson Sampling acts greedily under one sampled model. BAMCP earns positive discounted return without free examples and adapts risk to shared structure, but results are task-specific.
Key insights
- Formal counterexamples illustrate that myopic planning strategies like Thompson Sampling are inherently over-optimistic for discounted objectives in temporally extended MDPs, leading to suboptimal risk-taking and lack of commitment.
- Rigorous Bayes-adaptive planning, even with approximations (like BAMCP), can effectively mitigate the risks of over-optimism, guiding safer and more efficient exploration in complex, uncertain environments.
- A Bayes-optimal agent's exploration strategy adaptively changes based on environmental factors, such as the cost of errors, the planning horizon (discount factor), and the inherent uncertainty in the environment (e.g., concentration parameter α of the CRP model).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.