Reproduce “Model-Free Episodic Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra and 1 more
Episodic control learns reinforcement-learning policies by storing the highest returns observed for state–action pairs and reusing values from nearby states. It was evaluated on Atari games and Labyrinth tasks. In Double T-Maze, it achieved positive expected reward after about 20–30 million frames, while A3C never exceeded zero. These benchmark results do not establish performance in arbitrary stochastic environments.
Key insights
- Biologically inspired episodic memory can significantly enhance the data efficiency of artificial learning systems, especially when combined with mechanisms for generalization.
- The 'risk-seeking' nature of storing the maximum historical return, coupled with backward replay of experiences, enables rapid exploitation of highly rewarding events.
- Instance-based learning, when paired with appropriate learned state representations (e.g., from VAEs) and k-nearest neighbor approximation, can effectively generalize in environments with high state novelty, moving beyond simple memorization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.