Reproduce “Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning” on OpenResearch

About this paper

Weicong Liang, Yuhui Yuan, Henghui Ding, Xiao Luo, Weihong Lin, Ding Jia, Zheng Zhang, Chao Zhang and 1 more

Researchers from Peking University, ETH Zurich, and Microsoft Research Asia introduced a non-parametric method to expedite large-scale Vision Transformers for dense prediction tasks by intelligently reducing and reconstructing token resolutions. This approach accelerates models like Segmenter+ViT-L/16 by 40% in FPS and saves 30% in GFLOPs on ADE20K, while maintaining over 99.5% of its original performance without requiring any fine-tuning.

Key insights

• Well-trained Vision Transformers' intermediate token representations contain substantial local spatial redundancy, allowing for compression without significant information loss.
• Preserving high semantic fidelity between original and reconstructed high-resolution features is essential for achieving effective fine-tuning-free acceleration.
• Non-parametric token manipulation layers can effectively expedite large-scale ViTs for complex dense prediction tasks without requiring expensive parameter updates.

Summary by alphaXiv.