Reproduce “Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Bennett, Nathan Kallus
A new framework, Proximal Reinforcement Learning, enables robust Off-Policy Evaluation in Partially Observed Markov Decision Processes despite unmeasured confounding. This approach provides a general identification theory and yields semiparametrically efficient estimators for evaluating target policy values in complex sequential decision-making scenarios.
Key insights
- The value of a target policy in a POMDP with unmeasured confounding can be identified as the expectation of a doubly robust score function, making it statistically tractable.
- Proximal Causal Inference principles can be extended to dynamic, multi-time-step settings by recursively applying them through backward induction to tackle sequential confounding.
- The conditional moment restrictions defining the necessary 'bridge functions' can be re-framed and estimated directly from observed logging data, enabling practical implementation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.