Reproduce “Foley-Omni: A Unified Multimodal Generation Model from Task-Level Audio Synthesis to Complete Video Soundtrack Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ye Tao, Lupeng Liu, Xuenan Xu, Jiasun Feng, Jiarui Wang, Ying Qin, Shuiyang Mao, Wei Liu and 1 more
Foley-Omni, a unified multimodal model, generates complete video soundtracks from video and structured text inputs, jointly synthesizing speech, sound effects, and music. This approach achieves superior speech intelligibility (7.59% WER), audiovisual consistency, and perceived quality on a new V2ST-Bench benchmark compared to compositional baselines.
Key insights
- Jointly generating all audio components (speech, sound effects, music) within a shared latent space results in more coherent and higher-quality complete video soundtracks compared to combining outputs from separate expert systems.
- A hybrid condition injection strategy, utilizing both cross-attention for semantic video features and an additive path for time-aligned synchronization features, is critical for achieving precise audiovisual alignment and consistency.
- Employing a curriculum learning strategy is crucial for successfully training unified models that perform heterogeneous generation tasks, as it mitigates destructive task interference and preserves individual component quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.