Reproduce “Music Grounding by Short Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zijie Xin, Minquan Wang, Jingyu Liu, Ye Ma, Quan Chen, Peng Jiang, Xirong Li
Researchers from Renmin University of China and Kuaishou Technology define Music Grounding by Short Video (MGSV), a task to localize a music moment for a given video, and introduce MaDe, a unified deep network that achieves an mIoU of 0.722 in single-music grounding and an 8.3% MoR1 in music-set grounding. The work also establishes a real-world benchmark dataset, MGSV-EC, with 53,194 short videos.
Key insights
- Traditional cross-attention (CA) based multimodal feature fusion, effective in video-text tasks, is often detrimental for video-music grounding, suggesting distinct interaction dynamics are needed.
- Initializing the moment detection decoder with the video embedding provides essential query-modality context from the outset, leading to improved moment localization.
- A combination of simple mean-pooled and video-conditioned X-Pool music embeddings for matching enhances the model's ability to capture diverse aspects of music relevance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.