Reproduce “Unified Pretraining Target Based Video-music Retrieval With Music Rhythm And Video Optical Flow Information” on OpenResearch

About this paper

Tianjun Mao, Shansong Liu, Yunxuan Zhang, Dian Li, Ying Shan

A novel framework enhances video-music retrieval by employing a unified target label set for pretraining and integrating clip-level music rhythm and video optical flow features. This approach achieved superior Recall@K metrics, with up to 8.82% Recall@1 on an internal dataset, and was preferred by 50% of human evaluators over existing baselines.

Key insights

• Unifying the pretraining target label sets for video and music feature extractors effectively bridges the inherent "modality gap," leading to more compatible cross-modal embeddings.
• Integrating fine-grained temporal information, including clip-level sequential embeddings and explicit cues like music rhythm and video optical flow, is crucial for capturing dynamic video-music correlations.
• The use of a Transformer-based architecture with cross-modal attention is effective in fusing information from both modalities, improving their alignment and matching performance.

Summary by alphaXiv.