Reproduce “A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, Yang You
A training-free method accelerates large Vision-Language Models (VLMs) by leveraging a compact VLM to precisely guide visual token pruning and enable early exiting for "easy" tasks. This approach achieves up to 91% visual token pruning while retaining over 89% of original performance, significantly enhancing inference efficiency across various VLM architectures and benchmarks.
Key insights
- Partial attention information from single layers is insufficient for accurately identifying critical visual tokens in large VLMs, especially when aggressively pruning, leading to substantial performance drops.
- Global attention information, aggregated across all layers of a VLM, is significantly more effective for preserving essential visual tokens and maintaining performance even at very low retention ratios.
- The global attention map aggregated from a small VLM closely mirrors that of a large VLM, enabling the small VLM to serve as an efficient and precise proxy for guiding large VLM token pruning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.