Reproduce “Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daniel De Dios Allegue, Jinke He, Frans A. Oliehoek

Researchers at Delft University of Technology developed structured attention mechanisms for Transformer-based world models in partially observable reinforcement learning, aiming to improve sample efficiency by prioritizing informative agent history. Their Gaussian Attention variant achieved a 77% relative increase in mean human-normalized score and a 100% relative increase in median human-normalized score on the Atari 100k benchmark.

Key insights

  • Encoding structured temporal priors into Transformer self-attention significantly enhances sample efficiency and dynamics modeling in partially observable RL.
  • Smooth distributional priors (Gaussian Attention) are more effective and robust for capturing temporal dependencies than rigid memory-length priors (Adaptive Attention).
  • The proposed attention priors introduce negligible computational overhead while substantially improving performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.