Reproduce “BiFormer: Vision Transformer with Bi-Level Routing Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, Rynson Lau
Researchers from City University of Hong Kong and SenseTime Research developed BiFormer, a Vision Transformer that uses Bi-level Routing Attention to achieve dynamic, query-aware sparse attention. This architecture offers improved performance-computation trade-offs across image classification, object detection, and semantic segmentation, with BiFormer-S achieving 83.8% top-1 accuracy on ImageNet-1K.
Key insights
- Vanilla self-attention's quadratic complexity limits Vision Transformers, while existing efficient methods are either statically sparse or data-adaptive but lack query-specific awareness.
- Intelligently filtering irrelevant key-value pairs at a coarse region level, followed by fine-grained token-to-token attention, enables dynamic and query-adaptive sparsity.
- By gathering relevant key/value pairs into contiguous memory, dense matrix multiplications can be used, ensuring the theoretical efficiency gains of dynamic sparsity translate into practical GPU performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.