Reproduce “Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Palenicek, Florian Vogt, Joe Watson, Jan Peters
Researchers from TU Darmstadt, University of Freiburg, and University of Oxford developed an enhanced off-policy reinforcement learning algorithm, CrossQ + WN, that integrates Batch Normalization with Weight Normalization. This approach enables reliable scaling to higher Update-to-Data ratios, achieving competitive sample efficiency on challenging DeepMind Control Suite and MyoSuite tasks without requiring complex algorithmic interventions.
Key insights
- The instability of vanilla CrossQ at high UTDs stems from rapidly growing L2 norms of the critic network parameters, leading to a decreased effective learning rate and loss of network plasticity.
- Weight Normalization effectively counteracts these growing weight norms, maintaining a more stable effective learning rate without altering the critic's output due to the scale-invariance provided by Batch Normalization.
- Combining Batch Normalization and Weight Normalization creates a synergistic effect that stabilizes training dynamics at high UTDs, allowing for consistent sample efficiency improvements without complex interventions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.