Reproduce “AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

William Chen, Prem Seetharaman, Rithesh Kumar, Oriol Nieto, Shinji Watanabe, Justin Salamon, Zeyu Jin

AudioChat edits and generates layered audio scenes while using the original audio as context. It represents speech and other sounds as separate sources and combines text reasoning with continuous audio generation. On simulated StoryGen-Eval, it receives human ratings of 3.92 ± 0.05 for consistency and 3.12 ± 0.06 for instruction following, versus 1.76 ± 0.05 and 2.28 ± 0.06 for a cascade baseline; this does not establish performance on arbitrary real-world edits.

Key insights

  • Synthetically generated data from an LLM agent can effectively bridge the data scarcity gap for complex audio stories, enabling the training of robust multimodal foundation models.
  • A unified, end-to-end multimodal LLM architecture with self-cascaded layers can effectively integrate audio understanding, generation, and open-ended editing within a single model.
  • The Audio Transfusion Forcing objective, especially with Diffusion Forcing, is crucial for ensuring the model genuinely performs complex edits rather than simply copying input audio, leading to improved fidelity and instruction adherence.
  • Novel task-specific metrics like editFLAM and ∆multiFLAM are essential for accurately evaluating nuanced performance in complex audio storytelling and editing, showing strong correlation with human judgment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.