Reproduce “Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process” on OpenResearch

About this paper

Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, Rui Song

This research introduces a framework for off-policy confidence interval estimation in infinite horizon reinforcement learning with unmeasured confounders, leveraging auxiliary mediator variables. The proposed COPE estimator achieves semiparametric efficiency and provides valid confidence intervals, outperforming existing methods by addressing unmeasured confounding in both simulations and a real-world ride-hailing application.

Key insights

• The value of a target policy can be identified from observational data even in infinite horizon settings with unmeasured confounders, provided certain mediator variables and causal assumptions are met.
• The proposed COPE estimator is doubly robust, meaning it remains consistent if either specific Q-function/marginal action distributions or specific marginal density ratios/mediator distributions are correctly estimated.
• COPE achieves the semiparametric efficiency bound, indicating that it makes optimal use of the available data for precise estimation and valid uncertainty quantification.

Summary by alphaXiv.