Reproduce “Episodic Curiosity through Reachability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly
Researchers at Google Brain, DeepMind, and ETH Zurich developed Episodic Curiosity (EC), an intrinsic reward mechanism defining novelty by how many environment steps are needed to reach a current observation from past experiences stored in episodic memory. This approach effectively drives exploration in sparse reward environments while avoiding unproductive "couch-potato" behaviors seen in other curiosity methods.
Key insights
- Defining novelty based on the "effort" (environment steps) required to reach a state from previously visited states leads to more productive exploration than methods based on mere unpredictability.
- Leveraging an episodic memory allows the agent to contextualize new observations against its specific past experiences within an episode, driving the agent towards genuinely new regions.
- Integrating a learned reachability predictor with an episodic memory effectively mitigates the "couch-potato" problem, where agents get stuck on easily controllable but uninformative sources of surprise.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.