Reproduce “Geometry-Guided 3D Visual Token Pruning for Video-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Li, Zehao Huang, Jiahui Fu, Naiyan Wang, Si Liu
The Geo3DPruner framework from Beihang University and Zhongguancun Academy enhances the inference efficiency of video-language models for 3D scene understanding by introducing a geometry-guided visual token pruning method. It achieves over 90% performance retention while pruning 90% of visual tokens by leveraging multi-view consistency and spatial diversity across 3D spatial videos.
Key insights
- Leveraging global cross-frame geometric relevance, modeled through attention mechanisms, is crucial for effectively identifying and removing multi-view redundancy in 3D spatial videos.
- A combined two-stage pruning approach, addressing both local view consistency and global spatial diversity, is more effective than single-stage methods for preserving performance and comprehensive scene representation in 3D.
- Strategic visual token pruning can not only enhance computational efficiency but also potentially improve model performance by removing redundant or noisy visual tokens, leading to more compact and informative representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.