Reproduce “On Instrumental Variable Regression for Deep Offline Policy Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yutian Chen, Liyuan Xu, Caglar Gulcehre, Tom Le Paine, Arthur Gretton, Nando de Freitas, Arnaud Doucet

The paper studies offline policy evaluation: deciding whether to deploy a reinforcement-learning policy using logged experience alone. It explains how sampled next-state randomness can correlate a regression input with its error, and examines instrumental-variable methods alongside FQE’s fixed target. In a simple stochastic MDP, LSTD-Q and FQE track truth while DBRM is biased; benchmarks show no universal winner.

Key insights

  • Minimizing the Bellman error in Q-function estimation intrinsically constitutes a regression problem subject to confounding.
  • The 'target network trick,' prevalent in deep RL, effectively addresses this confounding by stabilizing the target, thereby making the regression unbiased.
  • Instrumental Variable regression provides a robust theoretical framework for understanding and solving the confounding problem in offline Q-function estimation, unifying diverse OPE approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.