Reproduce “Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation” on OpenResearch

About this paper

Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-Lobato, Kun Zhang, Bernhard Schölkopf

CounTerfactual Reinforcement Learning (CTRL) introduces a framework for sample-efficient and personalized reinforcement learning by augmenting data using Structural Causal Models and counterfactual reasoning. The method surpasses existing model-based RL approaches on control tasks, particularly with limited data, and demonstrated higher estimated Q-values than physician policies for sepsis treatment on the MIMIC-III dataset, with personalized policies showing further improvements.

Key insights

• Structural Causal Models (SCMs) offer a theoretically sound framework for generating counterfactual data, enabling robust and valid data augmentation beyond simple interpolation or model-based predictions.
• Explicitly learning the underlying causal mechanisms and individual-specific factors from observational data is essential for applying counterfactual reasoning in real-world Reinforcement Learning scenarios.
• Incorporating personalized latent variables within SCMs allows for the creation of tailored policies that outperform general policies in environments with inherent heterogeneity.

Summary by alphaXiv.