Reproduce “PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pierre-David Letourneau, Manish Kumar Singh, Hsin-Pai Cheng, Shizhong Han, Yunxiao Shi, Dalton Jones, Matthew Harper Langston, Hong Cai and 1 more

Qualcomm AI Research presents PADRe, a unifying polynomial attention framework designed to replace self-attention in Vision Transformers, achieving linear computational and memory costs. The method maintains or enhances model accuracy while delivering 11x to 43x speedups on GPUs and mobile NPUs by leveraging hardware-friendly operations, making high-performance ViTs more efficient for on-device deployment.

Key insights

  • Polynomial functions can effectively approximate complex attention mechanisms, enabling the design of inherently linear-cost and hardware-friendly alternatives to quadratic self-attention.
  • Designing attention with simple, parallelizable operations, such as Hadamard products and structured convolutions, directly translates to substantial real-world latency reductions on diverse hardware platforms like mobile NPUs.
  • The PADRe framework provides a principled, unifying perspective on various efficient attention mechanisms, allowing for systematic scaling of model capacity through polynomial degree while retaining linear computational complexity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.