Reproduce “EventSTU: Event-Guided Efficient Spatio-Temporal Understanding for Video Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhao Xu, Xin Dong, Yue Li, Haoyuan Shi, Zhiwei Xiong
EventSTU, developed by the University of Science and Technology of China, is a training-free framework that leverages event-guided spatio-temporal understanding to enhance Video Large Language Models. It achieves a 3.01x FLOPs reduction and 3.10x prefilling speedup while improving performance on long video understanding tasks, applicable to both real and simulated event data.
Key insights
- Biological vision principles and event-camera properties (change-triggered, sparse, salient) offer a potent pathway to enhance computational efficiency in Video-LLMs.
- A holistic approach combining temporal keyframe selection and spatial token pruning, both guided by event data and question relevance, effectively addresses video redundancy.
- Reusing the vision encoder's attention matrix for semantic-aware token pruning enables a 'zero-cost' approach that minimizes computational overhead compared to dedicated pruning methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.