Reproduce “Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2” on OpenResearch

About this paper

Zachary Daniels, Aswin Raghavan, Jesse Hostetler, Abrar Rahman, Indranil Sur, Michael Piacentino, Ajay Divakaran

Researchers at SRI International developed a model-free generative replay approach for lifelong reinforcement learning, employing a wake-sleep training cycle and a novel Hidden Replay architecture that operates on latent feature representations. This method successfully mitigates catastrophic forgetting and significantly improves sample efficiency, achieving 80-90% of expert performance in StarCraft II minigames with only 6% of the samples required by a single-task expert.

Key insights

• Replaying independently sampled observation-action or feature-action pairs is sufficient to mitigate forgetting in LRL, simplifying generative replay compared to trajectory-based methods.
• Operating generative replay on latent feature representations (Hidden Replay) is more effective than generating raw observations, as it simplifies the generative modeling task and makes the policy less sensitive to imperfect decoders.
• A small, diverse random replay buffer (RaR) is crucial for preventing feature drift and stabilizing the feature extractor, significantly improving lifelong learning performance and catastrophic forgetting mitigation, especially when combined with experience replay.

Summary by alphaXiv.