Reproduce “Foundations of Multivariate Distributional Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Mark Rowland
The paper develops tractable methods for learning multivariate return distributions from accumulated reward signals. It uses particle-based and categorical finite representations, with a signed-measure relaxation that makes categorical temporal-difference updates affine and provably convergent in tabular settings under sampling assumptions. Categorical methods improve with atom count and outperform EWP-TD in tests, while EWP-TD convergence remains unproved.
Key insights
- The multivariate distributional Bellman operator is a contraction in the MMDκ metric, providing a robust theoretical foundation for the convergence of value iteration in MDRL.
- The crucial insight that categorical MMD projections are non-affine for d ≥ 2 led to the novel strategy of relaxing to mass-1 signed measures, which enables an affine projection suitable for stochastic approximation in TD learning.
- Different distribution representations (particle-based vs. categorical) exhibit distinct approximation error scaling with reward dimension d and number of approximating elements m, informing their practical utility tradeoffs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.