Reproduce “OVG-HQ: Online Video Grounding with Hybrid-modal Queries” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Runhao Zeng, Jiaqi Mao, Minghao Lai, Minh Hieu Phan, Yanjie Dong, Wei Wang, Qi Chen, Xiping Hu

Researchers from Shenzhen MSU-BIT University and the University of Adelaide introduce Online Video Grounding with Hybrid-modal Queries (OVG-HQ), a task enabling real-time video segment localization using combined text, image, and video queries. Their OVG-HQ-Unify framework, featuring a Parametric Memory Block and a hybrid distillation strategy, outperforms existing online and adapted offline methods, achieving 45.95 FPS on an RTX 4090 GPU while improving oR^1_{0.5} by up to 8.98% for certain query types on their new QVHighlights-Unify dataset.

Key insights

  • The Parametric Memory Block (PMB) offers an expressive and efficient way to retain historical context in streaming scenarios by dynamically updating network parameters, outperforming LSTMs and self-attention.
  • Unifying training for hybrid-modal queries without specific strategies can lead to modality imbalance, where stronger modalities (like rich text queries) overshadow weaker ones.
  • Dynamic updates of PMB parameters during inference time are crucial for adapting to unseen data and leveraging continuously accumulated historical information, leading to improved performance compared to fixed parameters.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.