Reproduce “Music2Video: Automatic Generation of Music Video with fusion of audio and text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yoonjeon Kim, Joel Jang, Sumin Shin
KAIST researchers developed MUSIC2VIDEO, a framework that automatically generates music videos by intelligently fusing audio and text (lyrics) inputs. The system produces visually coherent videos with dynamic scene transitions aligned with musical structure and maintains temporal consistency between frames.
Key insights
- A segment-based, intelligent switching strategy for multi-modal guidance (text or audio) effectively prevents conflicts and leverages the strengths of each modality.
- Analyzing musical features like onset strength and intensity provides a robust and automatic way to dynamically segment music and control video pacing.
- Averaging CLIP prompt vectors from consecutive frames proved a simple yet effective method for achieving temporal consistency and object persistence in generated video.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.