Reproduce “Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences” on OpenResearch

About this paper

Aosong Feng, Irene Li, Yuang Jiang, Rex Ying

The Diffuser model from Yale University introduces a multi-hop attention diffusion mechanism, implemented via Personalized PageRank, and optimized sparse attention patterns to efficiently process long sequences. This approach achieves state-of-the-art performance across various tasks while providing substantial memory savings and improved robustness over existing efficient Transformer architectures.

Key insights

• Graph diffusion processes, like Personalized PageRank, can effectively model multi-hop token correlations in a computationally efficient manner, enabling a broader receptive field within a single attention layer.
• Designing sparse attention patterns with good expander graph properties, particularly through token-wise global and random connections, is crucial for approximating the expressiveness of full attention with fewer connections.
• The model is a universal sequence approximator proven to achieve contextual mappings with fewer layers than sparse attention models without diffusion, due to its expanded receptive field.

Summary by alphaXiv.