Reproduce “Provably Efficient Causal Reinforcement Learning with Confounded Observational Data” on OpenResearch

About this paper

Lingxiao Wang, Zhuoran Yang, Zhaoran Wang

Researchers at Northwestern and Princeton Universities developed DOVI and DOVI+, algorithms that provably improve the sample efficiency of online reinforcement learning by effectively incorporating confounded observational data. The method quantifies this benefit through an information gain factor, demonstrating that regret can become arbitrarily small with sufficient informative observational data under specific causal identification criteria.

Key insights

• Causal identification principles, specifically the backdoor and frontdoor criteria, can be systematically embedded within RL algorithms to correctly infer causal effects from confounded observational data.
• By constructing 'backdoor-adjusted' or 'frontdoor-adjusted' features, observational data can effectively regularize or 'warm-start' value function estimations, leading to provably unbiased learning.
• The information gain derived from properly adjusted observational data can be rigorously quantified, leading to tighter uncertainty estimates and consequently, significantly reduced online exploration needs.

Summary by alphaXiv.