Reproduce “AudioX: A Unified Framework for Anything-to-Audio Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyue Tian, Zhaoyang Liu, Yizhu Jin, Ruibin Yuan, Liumeng Xue, Xu Tan, Qifeng Chen, Wei Xue and 1 more
Researchers from Hong Kong University of Science and Technology developed AudioX, a unified framework for generating audio and music from diverse multimodal inputs (text, video, audio), accompanied by the large-scale IF-caps dataset. The system achieved state-of-the-art performance across various generation tasks and demonstrated superior fine-grained instruction following in text-to-audio generation.
Key insights
- A 'cross-modal regularization effect' was observed, where high-quality textual supervision significantly improves modality alignment and overall performance across all conditioning modalities, including video-to-audio.
- The Multimodal Adaptive Fusion (MAF) module, with its gating and query-based cross-attention mechanisms, is crucial for optimal multimodal fusion, reducing interference, and enhancing generation quality.
- High audio fidelity in generative models does not consistently correlate with strong instruction-following capabilities; specialized training with fine-grained data is necessary for precise control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.