Reproduce “AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiwu Zhong, Zhuoming Liu, Yin Li, Liwei Wang
The AIM method introduces a training-free framework that combines within-frame token merging and progressive pruning across LLM layers to enhance the inference efficiency of Multi-Modal Large Language Models (MLLMs). It achieves up to an 8.0x reduction in prefill time for video LLMs and 2.7x for image LLMs with minimal performance degradation, additionally enabling enhanced long video understanding by accommodating more input frames.
Key insights
- Significant redundancy exists within visual tokens generated by MLLM encoders, allowing for substantial reduction through merging and pruning with minimal impact on performance.
- LLM layers exhibit distinct roles in multimodal processing; early layers are crucial for cross-modal fusion and sensitive to visual token pruning, while later layers can tolerate more aggressive pruning as reasoning shifts towards textual content.
- A flexible pruning scheduler and tunable merging ratios enable adaptive inference, allowing MLLMs to dynamically adjust to a broad spectrum of accuracy-efficiency trade-offs based on computational constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.