Reproduce “WavJourney: Compositional Audio Creation with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Meng Cui, Qiushi Huang, Jinhua Liang, Yin Cao and 3 more
WavJourney turns text instructions into editable audio plans that coordinate speech, music, and sound effects generated by specialist models. An LLM writes a structured script, which a fixed compiler executes and mixes without additional model training or fine-tuning. On AudioCaps, WavJourney scores 3.75 OVL and 3.74 REL, exceeding AudioGen and AudioLDM on OVL but not ground truth on REL.
Key insights
- Orchestrating specialized audio generation models with an LLM provides an effective and modular paradigm for compositional audio creation, circumventing the need for a single, monolithic end-to-end model.
- Using a structured audio script as an intermediate representation enhances interpretability, improves stability during inference, and mitigates potential errors (hallucinations) compared to LLMs directly generating complex program code.
- Integrating high-quality zero-shot Text-to-Speech models, when guided by LLM contextual understanding, can significantly elevate the subjective perception of speech-inclusive audio, sometimes surpassing ground truth recordings in perceived quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.