Reproduce “RUDDER: Return Decomposition for Delayed Rewards” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, Sepp Hochreiter
This paper presents RUDDER, a method that addresses the long-term credit assignment problem in reinforcement learning by transforming delayed rewards. It redistributes rewards to make expected future rewards zero, reformulating the task into a regression problem, and achieves exponential speed-ups on artificial delayed reward tasks and improves performance on Atari games.
Key insights
- Transforming an MDP into a "return-equivalent" Sequence-Markov Decision Process (SDP) allows for a reward redistribution where expected future rewards become zero, simplifying Q-value estimation to immediate rewards.
- Optimal reward redistribution can be achieved by decomposing the total return of a sequence and assigning contributions to individual state-action pairs, effectively turning the credit assignment problem into a regression task.
- Long Short-Term Memory (LSTM) networks are effective for learning to predict sequence-wide returns and then attributing these predictions back to specific actions, leveraging their ability to capture long-term dependencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.