Reproduce “A Principled Path to Fitted Distributional Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sungee Hong, Jiayi Wang, Zhengling Qi, Raymond K. W. Wong
A principled framework for Fitted Distributional Evaluation (FDE) was developed, identifying functional Bregman divergences as theoretically sound discrepancy measures for distributional Off-Policy Evaluation. This approach consistently outperformed existing baselines like Fitted Likelihood Evaluation, QRDQN, and IQN on Linear Quadratic Regulators and Atari game benchmarks, exhibiting improved accuracy and stability.
Key insights
- Functional Bregman divergences are uniquely suited as discrepancy measures for FDE, as the true distributional Bellman operator's output is their unique minimizer for the population objective function.
- The research rigorously defined and analyzed two classes of contraction-inducing metrics (supremum-extended and expectation-extended) for the distributional Bellman operator, essential for convergence guarantees.
- A systematic framework allows for deriving and validating a wide range of discrepancy measures for FDE, some of which were previously unexplored in distributional OPE, offering options for unbounded and multi-dimensional return distributions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.