Reproduce “Representative Attention For Vision Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuntong Li, Hainuo Wang, Hengxing Liu, Mingjia Li, Xiaojie Guo

Representative Attention (RPAttention) from Tianjin University introduces a "representation-driven" global attention mechanism for Vision Transformers, which efficiently processes visual information by dynamically forming a compact set of semantically meaningful representative tokens. The method achieves linear computational complexity (O(NM + M^2)) with respect to spatial tokens while delivering consistent performance gains, such as a 4.0% Top-1 accuracy increase on PVT-Tiny for ImageNet-1K and improved average precision for COCO object detection.

Key insights

  • Aggregating visual tokens based on their semantic feature similarity, rather than fixed spatial coordinates, enables more coherent and contextually rich global representations.
  • A compact set of learned representative tokens can effectively mediate global information exchange, reducing computational complexity from O(N^2) to O(NM + M^2) while preserving a global receptive field.
  • The EM-inspired, single-step gather mechanism provides an efficient, end-to-end learnable way to perform semantic routing without requiring iterative refinement common in other object-centric approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.