Reproduce “Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data” on OpenResearch

About this paper

Danyang Wang, Chengchun Shi, Shikai Luo, Will Wei Sun

Researchers from Purdue University, London School of Economics, and ByteDance developed PESCAL, an offline reinforcement learning algorithm designed to learn optimal policies from observational data concurrently affected by unmeasured confounding and limited data coverage. This method, which uses a mediator variable and a novel pessimistic principle, achieved an approximate 8% improvement in online discounted returns in a real-world ride-hailing application compared to existing baselines.

Key insights

• The M2DP framework offers a principled way to integrate causal inference, specifically the front-door criterion, into dynamic sequential decision-making to handle unmeasured confounding.
• Applying pessimism to the mediator distribution provides a more tractable and practical approach to uncertainty quantification compared to quantifying uncertainty for the entire Q-function.
• PESCAL relaxes the stringent positivity assumption required by many offline RL methods, necessitating sufficient data coverage only for the optimal policy rather than all possible state-action pairs.

Summary by alphaXiv.