Reproduce “SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, Li Zhang
Researchers from Fudan University and Huawei's Noah’s Ark Lab developed SlowFocus, a mechanism for Video Large Language Models that enhances fine-grained temporal understanding by intelligently focusing high-frequency sampling on query-relevant video segments. It achieved significantly higher mIoU for temporal grounding and accuracy for temporal reasoning on the new FineAction-CGR benchmark while maintaining competitive performance on general video tasks.
Key insights
- Intelligent, query-driven mixed-frequency sampling can effectively resolve the token trade-off dilemma in Vid-LLMs, enabling fine-grained temporal understanding without a proportional increase in computational cost.
- Explicitly modeling temporal relationships through a dedicated temporal encoder and integrating multi-frequency information via the MMA module are crucial for accurate temporal grounding and reasoning.
- A progressive, multi-stage training approach is essential for successfully adapting Vid-LLMs to integrate adaptive sampling mechanisms and enhance specialized fine-grained temporal capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.