Reproduce “HFS: Holistic Query-Aware Frame Selection for Efficient Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiqing Yang, Yun Li, Daiqing Qi, Lehan Yang, Tianlong Wang, Wenhao Zhang, Sheng Li, Kin-man Lam
HFS improves long-video question answering by selecting a set of frames for a question and training the selector jointly with the video reasoner. It uses question-conditioned queries and a differentiable set objective that rewards relevance and coverage while penalizing temporal redundancy. On LongVideoBench validation, InternVL3-8B with HFS scores 60.2% versus 59.3% for MDP³ at 16 frames, with an additional 1.5B selector.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.