Reproduce “Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dingkun Zhang, Shuhan Qi, Yulin Wu, Xinyu Xiao, Xuan Wang, Long Chen
The DualSpeed framework introduces a fast-slow training paradigm for Multimodal Large Language Models, efficiently integrating visual token pruning to achieve substantial training speedups. This approach effectively resolves the training-inference mismatch, maintaining near-baseline performance on full visual inputs.
Key insights
- The training-inference mismatch, primarily affecting the LLM's self-attention, is the core barrier preventing VTP from being directly used in MLLM training.
- A dual-mode training strategy combined with a mode isolator and self-distillation can effectively bridge this mismatch, allowing VTP benefits during training without compromising inference performance.
- Up to 90% of visual tokens are redundant for MLLM training, indicating significant potential for efficiency gains by pruning visual inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.