Reproduce “Efficient Vision-Language Reasoning via Adaptive Token Pruning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xue Li, Xiaonan Song, Henry Hu
Adaptive Token Pruning (ATP) offers a training-free method to enhance Vision-Language Model (VLM) efficiency by dynamically pruning visual tokens. This approach reduces inference FLOPs by approximately 40% and achieves 1.5x speedups in latency with less than 1% accuracy loss across VQAv2, GQA, and COCO Captioning, while also indicating improved robustness to input perturbations.
Key insights
- Substantial VLM efficiency gains in FLOPs reduction and latency speedup can be achieved through dynamic visual token pruning without requiring any model retraining or architectural modifications.
- A hybrid scoring approach, combining context-independent visual saliency with text-driven relevance, effectively identifies and prioritizes semantically critical visual information.
- Adaptive token pruning not only improves efficiency but also shows early indications of enhancing VLM robustness against common visual corruptions and linguistic perturbations by focusing on salient features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.