Reproduce “Enrich and Detect: Video Temporal Grounding with Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shraman Pramanick, Effrosyni Mavroudi, Yale Song, Rama Chellappa, Lorenzo Torresani, Triantafyllos Afouras
A multimodal LLM framework named ED-VTG combines dynamic query enrichment with a lightweight temporal decoder to improve video temporal grounding. This approach achieves state-of-the-art performance across various grounding tasks, outperforming prior LLM-based and many specialist models, particularly in challenging zero-shot generalization scenarios.
Key insights
- Dynamically enriching imprecise natural language queries with video-derived context before temporal localization significantly improves grounding accuracy, outperforming methods relying solely on original queries or offline data augmentation.
- The Multiple Instance Learning (MIL) framework effectively mitigates noise from pseudo-labeled enriched queries by backpropagating loss only from the better of two predictions (original vs. enriched query), leading to robust learning.
- Combining the generalization and descriptive power of multimodal LLMs with a dedicated, lightweight regression-based temporal interval decoder successfully addresses the trade-off between semantic understanding and precise temporal localization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.