Reproduce “HawkEye: Training Video-Text LLMs for Grounding Text in Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, Dongyan Zhao
HawkEye enhances video-text Large Language Models with robust temporal video grounding abilities by introducing InternVid-G, a large-scale automatically constructed dataset, and a novel coarse-grained recursive grounding mechanism. This framework achieves state-of-the-art zero-shot performance on temporal grounding tasks while preserving performance on general video question answering.
Key insights
- Coarse-grained textual representations of video segments (e.g., "beginning," "middle," "end") are significantly easier for LLMs to learn and utilize for temporal grounding than precise numerical timestamps.
- Recursive grounding, when combined with coarse-grained representations, enables LLMs to achieve fine-grained temporal localization iteratively, mimicking a binary search strategy.
- Automatically constructing a large-scale dataset like InternVid-G can effectively address the data scarcity problem for training temporal video grounding models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.