Reproduce “Guided Exploration with Proximal Policy Optimization using a Single Demonstration” on OpenResearch

About this paper

Gabriele Libardi, Gianni De Fabritiis

Researchers from Universitat Pompeu Fabra developed PPO+D, an algorithm that solves challenging 3D exploration problems with sparse rewards by leveraging a single human demonstration within the Proximal Policy Optimization (PPO) framework. This method consistently outperforms baselines requiring many demonstrations across complex tasks, making advanced deep reinforcement learning more applicable to real-world scenarios with limited expert data.

Key insights

• Effectively integrating even a single human demonstration into an on-policy algorithm like PPO can dramatically improve performance and data efficiency in complex, hard-exploration tasks where pure RL methods fail.
• Prioritizing the replay of unsuccessful trajectories based on their maximum estimated value is crucial for mitigating catastrophic forgetting and enabling the agent to combine learned sub-behaviors, especially in more challenging exploration scenarios.
• A dynamic sampling strategy that balances guidance from demonstrations with self-exploration and successful past experiences allows for a robust and incremental learning process.

Summary by alphaXiv.