Reproduce “QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyunwoo Oh, Hanning Chen, Sanggeon Yun, Yang Ni, Wenjun Huang, Tamoghno Das, Suyeon Jang, Mohsen Imani

QUILL, an algorithm-architecture co-design, improves the hardware efficiency of Multi-Scale Deformable Attention (MSDeformAttn) in Deformable Transformers. It achieves up to 7.29x higher throughput and 47.3x better energy efficiency compared to high-end GPUs, by transforming irregular memory accesses into cache-friendly, single-pass computation, while maintaining model accuracy.

Key insights

  • The primary bottleneck for MSDeformAttn on hardware stems from cache misses, low arithmetic intensity, and bank conflicts caused by its irregular, sparse query patterns, rather than FLOP count alone.
  • Dynamically reordering spatially proximal queries at runtime can transform inherently irregular memory access into predictable, cache-friendly patterns, significantly enhancing data locality and memory energy efficiency.
  • Fusing the entire deformable attention pipeline into a single hardware pass dramatically reduces redundant memory accesses and increases arithmetic intensity, allowing algorithmic sparsity to translate directly into hardware efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.