Reproduce “Q-learning with Posterior Sampling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Priyank Agrawal, Shipra Agrawal, Azmat Azati

PSQL introduces a model-free Q-learning algorithm that directly applies Gaussian posterior sampling to Q-values for exploration in tabular episodic MDPs. This approach establishes a near-optimal ~O(H^2 * sqrt(SAT)) cumulative regret bound and provides a novel Bayesian interpretation for the Q-learning update rule, including the (H+1)/(H+n) learning rate.

Key insights

  • The Q-learning update rule, particularly its specific learning rate, can be naturally derived from a regularized Bayesian inference perspective, providing a deeper understanding of its mechanics.
  • Combining posterior sampling with bootstrapped Temporal Difference (TD) learning requires carefully engineering high-probability optimism in the value function target rather than directly in action selection.
  • Novel analytical techniques are needed to manage error accumulation in bootstrapped estimates and ensure optimism is maintained across multiple steps in an episode.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.