Reproduce “CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhuoran Song, Yihong Xu, Zhezhi He, Li Jiang, Naifeng Jing, Xiaoyao Liang

Researchers at Shanghai Jiao Tong University developed CP-ViT, a cascade pruning framework that dynamically identifies and prunes uninformative patches and attention heads in Vision Transformers (ViTs) to reduce computational redundancy. This method achieved up to 46% FLOPs reduction with less than 1% accuracy loss on ImageNet after fine-tuning, outperforming existing state-of-the-art pruning techniques.

Key insights

  • Not all patches and attention heads contribute equally to ViT performance; a significant amount of redundancy exists that can be exploited for structured pruning.
  • Tracking cumulative informativeness across layers, rather than just locally, is crucial for maintaining accuracy in cascade pruning scenarios by preserving features vital for downstream processing.
  • Dynamically adjusting pruning ratios based on a layer's attention range (clarity of interdependencies) leads to more effective and accurate compression than uniform pruning across all layers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.