Reproduce “Bayesian learning for the stochastic shortest path problem” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chon Wai Ho, Sumeetpal S. Singh, Jiaqi Guo
Bayes-BR infers uncertainty about long-run action values from Bellman equations, samples a plausible value function, and follows its greedy policy for an episode before updating its belief. A Gaussian-kernel relaxation makes its posterior sampleable. In depth-5 Deep Sea, regret is lower than for tested Bayes-TD variants at 0.01 and 0.1; all tested methods fail at 0.5 or higher.
Key insights
- The exact posterior distribution for Q* parameters in deterministic reward environments is characterized as a "manifold density" residing on a lower-dimensional manifold in the parameter space.
- A significant unidentifiability issue exists with the relaxed likelihood: for tabular Q* and improper policies, the likelihood can be invariant to translations, potentially biasing the posterior towards non-optimal, recurrent policies.
- The approximate posterior for tabular Q* models with Gaussian priors and kernels takes the form of a mixture of truncated-Gaussians, allowing for analytical calculation of optimal action probabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.