Reproduce “Model-Free and Model-Based Policy Evaluation when Causality is Uncertain” on OpenResearch

About this paper

David Bruns-Smith

Research identifies that the tractability of off-policy evaluation with unobserved confounders depends critically on whether confounders are IID across periods. The work develops both model-free and robust model-based methods for computing worst-case finite-horizon policy value bounds under this assumption, demonstrating that model-based approaches yield sharper estimates by leveraging domain knowledge.

Key insights

• The assumption that unobserved confounders are IID each period is the crucial factor determining the tractability of off-policy evaluation, as it simplifies the problem to an MDP over observed states.
• Integrating domain knowledge about transition confounding (Δ) into model-based robust MDPs produces substantially tighter and more useful lower bounds compared to model-free direct methods.
• Policy evaluation becomes significantly more difficult with persistent unobserved confounders, leading to highly conservative bounds, which identifies a major open challenge for future research.

Summary by alphaXiv.