Reproduce “Fast Vision Transformers with HiLo Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zizheng Pan, Jianfei Cai, Bohan Zhuang

Researchers at Monash University developed LITv2, an efficient Vision Transformer with a novel HiLo attention mechanism that processes high- and low-frequency image features separately. This architecture achieved substantial inference speedups (e.g., 13-36% higher throughput) and reduced memory consumption across image classification, object detection, and semantic segmentation tasks, while maintaining competitive accuracy.

Key insights

  • Disentangling and processing high and low-frequency components of visual features with specialized, hardware-friendly attention mechanisms significantly improves both computational efficiency and practical inference speed.
  • Directly optimizing for hardware throughput is a more effective design principle for Vision Transformers than solely relying on theoretical FLOPs, as evidenced by discrepancies between the two metrics.
  • Implicit positional encoding through depthwise convolutions can effectively replace explicit RPE, reducing overhead in dense prediction tasks and also enhancing the receptive field in early network stages.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.