Reproduce “Offline Reinforcement Learning with Causal Structured World Models” on OpenResearch

About this paper

Zheng-Mao Zhu, Xiong-Hui Chen, Hong-Long Tian, Kun Zhang, Yang Yu

FOCUS is an offline reinforcement learning algorithm that enhances generalization by incorporating causal structures into world models. It learns causal relationships from static observational data, demonstrating improved policy performance and robustness compared to models that disregard causality or adapt online causal discovery methods.

Key insights

• The theoretical framework establishes that generalization error bounds for model prediction and policy evaluation increase with spurious correlation strength and spurious variable density.
• Explicitly excluding spurious variables through causal modeling directly reduces the policy evaluation error bound, leading to improved generalization.
• Causal structures can be accurately and robustly inferred from purely observational offline data by adapting causal discovery techniques with an understanding of temporal dependencies.

Summary by alphaXiv.