Reproduce “Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jing Yi, Yaochen Zhu, Jiayi Xie, Zhenzhong Chen
Researchers from Wuhan University developed CMVAE, a cross-modal variational auto-encoder, to automatically recommend background music for micro-videos based on content. The model achieved superior Recall@K scores on the new TT-150k dataset, outperforming existing baselines, and proved robust to missing video textual information.
Key insights
- Forcing latent embeddings to generate features of the *other* modality (cross-generation) is effective for aligning video and music, particularly when associations in user-generated content are weak.
- Dynamically weighting multimodal video inputs using the Product-of-Experts principle, based on estimated variance, improves robustness by prioritizing more informative signals.
- Combining generative VAE components with a bi-directional ranking loss in a joint objective enhances the learning of matching patterns compared to either approach alone.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.