Reproduce “Adaptively Sparse Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gonçalo M. Correia, Vlad Niculae, André F.T. Martins

Adaptively Sparse Transformers enhance model interpretability and head specialization by replacing softmax with α-entmax and automatically learning the α parameter for each attention head. This approach allows heads to dynamically adjust their sparsity between dense and sparse attention, achieving comparable or slightly improved performance on neural machine translation tasks while revealing distinct attention specializations.

Key insights

  • Adaptive sparsity allows individual attention heads to dynamically adjust their focus, leading to diverse specializations (e.g., highly confident positional attention or novel subword-merging behaviors) that improve interpretability.
  • Automatically learning the α parameter through gradient-based optimization eliminates the need for manual hyperparameter tuning, making adaptive sparse attention practical and scalable for complex Transformer architectures.
  • Introducing adaptively sparse attention in Transformers can preserve or slightly improve neural machine translation performance while simultaneously enhancing head diversity and revealing clearer attention patterns.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.