Reproduce “Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kenny J. Young, Richard S. Sutton, Shuo Yang
An algorithm integrates episodic memory into a deep reinforcement learning agent, allowing it to efficiently learn and operate in environments with long-term dependencies. The approach avoids computationally expensive backpropagation through time by learning to selectively store and retrieve past states, demonstrating superior performance on a "secret informant problem" compared to recurrent neural networks.
Key insights
- Efficient online credit assignment to specific remembered states is possible by directly linking the utility of recalled information (via TD error) to the importance weight of the stored state, eliminating the need for BPTT.
- A tailored `O(n)` reservoir sampling algorithm can effectively manage a fixed-size, weighted episodic memory, allowing for dynamic learning of what states to retain.
- Learning to selectively store explicit past states with an importance-based write network allows RL agents to robustly handle long-term dependencies in non-Markovian environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.