Reproduce “The Primacy Bias in Deep Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, Aaron Courville
Researchers from Mila identified "primacy bias" as a key factor causing instability and poor generalization in deep reinforcement learning, where agents overfit to early experiences. They demonstrate that periodically re-initializing the final layers of an agent's neural networks, while preserving the replay buffer, consistently improves performance by up to 34% IQM on various benchmarks like Atari and DeepMind Control Suite, and enables stable training at previously problematic high replay ratios.
Key insights
- The "primacy bias" is a distinct form of overfitting in DRL where agents disproportionately overfit to early experiences, hindering subsequent learning and creating a self-reinforcing cycle of poor policy generation.
- The primacy bias is exacerbated by high replay ratios and longer N-step targets, leading to instability and poor performance in common DRL algorithms.
- A simple solution of periodically re-initializing the parameters of the last few neural network layers, while preserving the experience replay buffer, effectively mitigates primacy bias and improves learning stability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.