Reproduce “AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Changwoo Baek, Jouwon Song, Sohyeon Kim, Kyeongbo Kong
Researchers from Pusan National University and LG Electronics conducted an empirical study using effective rank and attention entropy to analyze visual token pruning in Large Vision-Language Models, revealing a trade-off between token diversity and hallucination. They developed AgilePruner, an adaptive pruning strategy that reduces FLOPs by 89% and mitigates hallucination while maintaining competitive accuracy across diverse benchmarks and LVLMs.
Key insights
- Diversity-oriented pruning preserves higher feature diversity but correlates with increased object hallucination, while attention-based pruning retains lower diversity and suppresses hallucination.
- Optimal pruning strategy is image-dependent: simple images benefit from attention-based pruning, whereas complex images are better served by diversity-based pruning.
- Image-aware adjustments to existing hybrid pruning methods, guided by image complexity (Erank), consistently improve performance and reduce hallucination.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.