Reproduce “Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anmin Wang, Nan Zhang, Wei Tao, Xiaoyang Qu, Guokuan Li, Jiguang Wan, Jianzong Wang
Researchers from Huazhong University of Science and Technology and Ping An Technology developed TRIAGE, a training-free, plug-and-play framework for Vision-Language Models (VLMs) that employs hierarchical visual budgeting to reduce computational costs in video reasoning. This approach consistently outperformed unoptimized baselines and other token compression methods across diverse benchmarks, frequently achieving higher accuracy with significantly fewer visual tokens.
Key insights
- A unified, hierarchical visual budgeting strategy effectively addresses both temporal and spatial redundancies in video, leading to improved efficiency and performance.
- Intelligent filtering of redundant visual information not only reduces computational load but can also enhance VLM reasoning by allowing the model to focus on salient cues.
- Combining diverse frame-level importance metrics (scene, motion, text relevance) with VLM-specific token-level cross-attention and diversity-aware selection is crucial for robust budgeting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.