Reproduce “Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, Aleksandr I. Panov
Forgetful Experience Replay (ForgER) improves the sample efficiency of deep reinforcement learning in complex, hierarchical environments by leveraging low-quality expert demonstrations through a structured replay buffer and a dynamic forgetting mechanism. The algorithm notably achieved the mining of a diamond in the challenging MineRL environment, outperforming previous competition solutions.
Key insights
- Dynamically reducing reliance on imperfect expert demonstrations (forgetting) is crucial for preventing overfitting to suboptimal behaviors and achieving superior performance in the long term.
- Structuring the replay buffer around automatically extracted subgoals from demonstrations enables effective hierarchical learning in complex, long-horizon tasks.
- Task-specific augmentation, by reusing common behaviors from other subtasks, significantly improves generalization and data efficiency in vision-based environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.