Reproduce “TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaewoo Lee, Keyang Xuan, Chanakya Ekbote, Sandeep Polisetty, Yi R. Fung, Paul Pu Liang
Researchers from MIT, UNC Chapel Hill, and other institutions developed TAMP, a token-adaptive layerwise pruning framework for Multimodal Large Language Models (MLLMs), which leverages unique multimodal token attributes for compression. This method achieved superior performance on LLaVA-NeXT and VideoLLaMA2, outperforming existing pruning techniques by up to 8.2 percentage points at high sparsity levels.
Key insights
- Output token distributions and their intra- and inter-modality diversities vary significantly across different layers, projection types, and modalities within MLLMs, indicating non-uniform parameter importance.
- Input activations in MLLMs can be biased towards modalities with larger token counts, which can lead to suboptimal pruning decisions if not accounted for.
- Value projection layers within Multi-Head Attention modules and intermediate MLLM blocks are particularly crucial for encoding and retaining rich multimodal information, requiring lower sparsity ratios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.