Reproduce “What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jinhwan Seo, Kyubeom Han, Jumin Lee, Junhyug Noh, Sung-eui Yoon

GroundFormer conditions video-frame features on question meaning before selecting an answer and its supporting interval. Communication tokens transfer candidate-question information to frames, while multiple-instance-learning attention links candidate scores to temporal evidence. On NExT-GQA, it reports Acc@GQA of 21.5 versus CRA-GQA’s 18.2, and question-pair PIoU of 28.2 versus 88.9. These benchmark results do not establish faithful localization in every case.

Key insights

  • Explicitly conditioning visual representations on question semantics early in the processing pipeline is critical for overcoming question-invariant grounding and enabling discriminative temporal localization.
  • A hierarchical multi-modal contrastive learning strategy, facilitated by a two-pass training pipeline, effectively aligns video, question, and answer embeddings for robust, question-specific reasoning.
  • Formulating Grounded VideoQA as a MIL problem with factorized cross-attention allows for simultaneous answer selection and implicit temporal evidence derivation under weak supervision.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.