Reproduce “V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinying Lin, Xuyang Liu, Yiyu Wang, Teng Ma, Wenqi Ren
V-CAST reduces visual tokens for video question answering by allocating a budget across frames according to neighboring semantic changes, then selecting patches within each frame while retaining original positions. With 32 frames and 25% retention, its benchmark-average score is 63.3 versus 64.2 uncompressed, and prefill latency falls 51.2% in the reported setup. It is training-free and evaluated under specified conditions.
Key insights
- Measuring semantic curvature of a video's temporal trajectory effectively identifies and prioritizes frames containing significant semantic transitions for token allocation.
- Preserving original spatio-temporal coordinates through pruning, rather than merging, is crucial for maintaining alignment with positional embeddings like MRoPE, thereby supporting robust spatio-temporal reasoning.
- A dual-anchor spatial selection strategy, balancing contextual diversity and feature activation, efficiently captures salient and varied visual evidence within individual frames.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.