Reproduce “Stabilizing Transformers for Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman and 5 more

Gated Transformer-XL (GTrXL) adapts Transformer-XL for partially observable reinforcement learning by moving layer normalization and replacing residual connections with gates. On DMLab-30, it reaches 117.6 ± 0.3 mean human-normalized return, versus 99.3 ± 1.0 for an LSTM trained with the same algorithm. The MERLIN comparison is not controlled, and GTrXL’s 100-capped score is slightly lower.

Key insights

  • Training instability in transformers for RL stems from factors like different optimization dynamics, sequence processing assumptions, and layer normalization placement.
  • Reordering layer normalization and introducing adaptive gating mechanisms are crucial for stabilizing transformer training in reinforcement learning.
  • Initializing gating mechanisms to behave like identity functions initially dramatically improves training speed and stability in RL contexts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.