Reproduce “Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alizée Pace, Hugo Yèche, Bernhard Schölkopf, Gunnar Rätsch, Guy Tennenholtz
The authors introduce Delphic ORL, an offline reinforcement-learning method that accounts for hidden confounding by comparing plausible world models consistent with observed records. It penalizes estimated action values where those worlds disagree, without identifying the hidden factor. In HiRID, its score was 62.2 ± 1.0 versus CQL’s 59.3 ± 0.9 with withheld-variable confounding; this is an off-policy estimate, not clinical evidence.
Key insights
- Hidden confounding creates a unique "delphic uncertainty" that reflects ambiguity across multiple causally distinct but observationally equivalent world models, and this uncertainty is not diminished by increased data quantity.
- Approximating the set of compatible world models through an ensemble of latent variable models allows for a practical quantification of nonidentifiable confounding bias.
- Incorporating delphic uncertainty into a pessimistic offline RL algorithm enables the learning of policies that are robust to hidden confounding by avoiding confounded actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.