Reproduce “Adaptive Attention Span in Transformers” on OpenResearch

About this paper

Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, Armand Joulin

Facebook AI Research introduced an adaptive attention span mechanism for Transformer models, which allows individual attention heads to dynamically learn their optimal context lengths. This approach yielded state-of-the-art performance on character-level language modeling tasks while significantly reducing computational and memory costs by allowing processing of contexts up to 8192 characters with a much smaller average span.

Key insights

• Different attention heads and layers within a Transformer exhibit specialized contextual needs, with lower layers focusing on short-range dependencies and higher layers attending to very long-range information.
• A universally long attention span is often unnecessary; the model can selectively attend to relevant context, significantly reducing computational overhead without sacrificing performance.
• Allowing for a very large potential context (e.g., 8192 characters) while learning smaller average effective spans (e.g., 245 characters) leads to better performance than fixed, shorter spans.

Summary by alphaXiv.