Reproduce “Unified Audio Generation and Editing via Joint Condition Modeling and Progressive Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haocheng Dong, Yuheng Lu, Cheng Gong, Shansong Liu, Xiao-Lei Zhang, Xuelong Li
AudioWeave unifies text-to-audio generation and instruction-based audio editing into a single model, utilizing a novel joint condition modeling and factorized position embedding with a progressive multistage training strategy. The model achieved the highest human-perceived quality and relevance scores for text-to-audio generation and consistently outperformed dedicated baselines across multiple audio editing tasks.
Key insights
- A unified Diffusion Transformer architecture can effectively handle both audio generation and complex instruction-based editing tasks without requiring task-specific components, simplifying system design.
- Factorized position embedding enables accurate distinction and integration of inter-sequence (text, target, reference audio) and intra-sequence positional information within a single model, accommodating diverse inputs.
- A progressive multistage training strategy effectively mitigates task competition and catastrophic forgetting in multi-task audio models, leading to enhanced overall performance and improved human-perceived quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.