Reproduce “Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yudi Zhang, Yali Du, Biwei Huang, Ziyan Wang, Jun Wang, Meng Fang, Mykola Pechenizkiy

The Generative Return Decomposition (GRD) framework introduces a causal approach to interpretable reward redistribution in reinforcement learning. It transforms sparse, delayed rewards into dense, causally-attributed proxy rewards, which leads to improved policy performance and a more transparent understanding of credit assignment across various MuJoCo tasks.

Key insights

  • The underlying generative process, including the causal structure, reward function, and dynamics, is identifiable from observable states, actions, and trajectory-wise long-term returns.
  • Learning a compact representation (`s_min`) of causally relevant state dimensions leads to more efficient and robust policy learning by filtering out irrelevant information.
  • Causal modeling provides a principled framework for intrinsically interpretable credit assignment, explaining *why* specific state-action pairs receive particular rewards.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.