Reproduce “Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mark Endo, Xiaohan Wang, Serena Yeung-Levy
Stanford University researchers developed FEATHER, a visual token pruning strategy for Vision-Language Models that achieves over 5x performance improvement on localization benchmarks compared to prior methods, maintaining high computational efficiency. The study also identifies positional biases in existing pruning approaches and reveals that many VLM benchmarks do not adequately assess fine-grained visual capabilities.
Key insights
- Early-layer attention-based pruning, like FastV, exhibits a strong positional bias, retaining tokens primarily from the bottom of the image due to the long-term decay property of RoPE.
- Many current VLM benchmarks do not adequately test fine-grained visual capabilities, allowing models with suboptimal pruning to achieve high scores and masking deficiencies in acceleration methods.
- Modifying pruning criteria by removing RoPE's influence and incorporating uniform sampling can effectively resolve positional bias and ensure better visual coverage in token selection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.