Reproduce “AdaTP: Attention-Debiased Token Pruning for Video Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengyuan Sun, Leqi Shen, Hui Chen, Sicheng Zhao, Jungong Han, Guiguang Ding
Tsinghua University researchers develop AdaTP, a training-free acceleration pipeline that addresses systematic attention biases in Video Large Language Models by introducing global and local debiasing modules to guide token pruning, achieving up to 73% reduction in computational costs (27.3% FLOPs vs vanilla models) while maintaining or improving performance on video understanding benchmarks like VideoMME and MLVU, with their approach discovering that effective bias correction can sometimes enhance model comprehension beyond the original unpruned versions.
Key insights
- Attention scores in Video LLMs exhibit systematic biases that undermine their utility for token importance assessment
- Text-visual alignment can effectively identify semantically relevant video segments for adaptive compression
- Removing spatial redundancy within temporally coherent segments preserves visual diversity while eliminating bias
- Effective debiasing can act as noise reduction, sometimes improving model comprehension beyond vanilla performance
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.