Reproduce “A Distributional Analogue to the Successor Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Marc G. Bellemare, Mark Rowland
The authors introduce a distributional successor measure (DSM) that learns a policy’s distribution of discounted future-state occupancies independently of a deterministic reward. Equally weighted model atoms are trained with two-level maximum mean discrepancy, allowing held-out rewards and criteria to be evaluated without retraining. In Windy Gridworld, mean and 0.4-CVaR favor different policies, a result demonstrated in two simulated domains.
Key insights
- A novel factorization (Proposition 3.2) demonstrates that the entire distribution of returns can be derived from the reward-agnostic Random Occupancy Measure.
- The distributional Bellman operator for ℸ_π is proven to be a γ-contraction mapping, ensuring the theoretical learnability and convergence of the Distributional Successor Measure.
- The δ-model effectively captures aleatoric uncertainty in future state occupancies, representing diverse possible 'futures' that an agent might experience.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.