Reproduce “Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, Pengtao Xie

The EViT framework introduces a method for Vision Transformers that progressively identifies and fuses less critical tokens during training using attention mechanisms. This approach enables ViTs to achieve up to a 50% increase in inference throughput with a minor 0.3% accuracy drop, or to improve recognition accuracy by 1.2% by processing higher-resolution inputs at a comparable computational cost, all without introducing additional model parameters.

Key insights

  • Vision Transformers contain redundant computations as a significant portion of image tokens are 'inattentive' and less critical for classification tasks.
  • Simply dropping inattentive tokens severely degrades accuracy, indicating the necessity of preserving some information from these tokens.
  • Weighted fusion of inattentive tokens into a single representation effectively mitigates information loss and maintains performance while significantly reducing computational load.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.