Reproduce “HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ping Gong, Jiawei Yi, Shengnan Wang, Juncheng Zhang, Zewen Jin, Ouxiang Zhou, Ruibo Liu, Guanbin Xu and 8 more

Researchers from USTC, Institute of Artificial Intelligence, Huawei Technologies, and Peking University developed HATA, a trainable and hardware-efficient hash-aware top-k attention mechanism for large language models. HATA re-conceptualizes top-k attention as an ordinal comparison problem, using learning-to-hash to select relevant keys, achieving up to 7.2x decoding speedup over dense attention while maintaining model accuracy on various benchmarks.

Key insights

  • For effective top-k attention, only the relative ordering of query-key scores is necessary, not their exact numerical values, which enables more efficient computational approaches.
  • Learning-to-hash can be effectively adapted to maintain relative query-key similarity with compact binary codes and bitwise operations, significantly reducing computational and memory overhead.
  • Integrating low-level hardware optimizations is crucial for translating theoretical efficiency gains of sparse attention into substantial, real-world inference speedups.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.