Reproduce “VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hengbo Xu, Shengjie Jin, Yanbiao Ma, Zhiwu Lu
VisionPulse introduces a training-free framework for dynamic visual token pruning in Large Multimodal Models, enabling them to adaptively manage visual context during multi-step reasoning. The approach achieves up to a 1.30x inference speedup and reduces reasoning trace length by over 11% while preserving near-baseline accuracy, by retaining as few as 5% of visual tokens per step.
Key insights
- The need for visual evidence in multimodal reasoning is dynamic and step-dependent, with the set of critical visual tokens evolving throughout the reasoning process.
- A strong linear correlation exists between the total visual attention mass at a given step and the number of actively used visual tokens, providing a reliable signal for dynamic budgeting.
- Maintaining redundant visual context not only incurs computational overhead but can also misdirect LMMs, leading to elongated and less reliable reasoning traces (the "coupled bottleneck").
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.