Reproduce “Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning” on OpenResearch

About this paper

Baiyuan Chen, Shinji Ito, Masaaki Imaizumi

Researchers from The University of Tokyo and RIKEN AIP show that transformers can achieve minimax optimal dynamic regret in non-stationary reinforcement learning, by learning to internally approximate advanced adaptive strategies like window schedulers and restart mechanisms through supervised pretraining. Empirical results in linear bandit environments demonstrate performance comparable to or exceeding specialized non-stationary algorithms.

Key insights

• Transformers can implicitly learn and execute sophisticated adaptive strategies for non-stationary environments, such as window-based data processing and restart policies, solely through in-context learning and pretraining.
• The transformer's hidden states can be conceptualized as maintaining multiple hypothetical policies, leveraging learned randomness in self-attention to sample among them for dynamic adaptation.
• Achieving optimal dynamic regret in non-stationary settings requires transformers of sufficient expressivity (small approximation error) and training on a sufficiently large and diverse dataset.

Summary by alphaXiv.