Reproduce “Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ge Zhu, Yutong Wen, Zhiyao Duan

Research by Zhu, Wen, and Duan from the University of Rochester provides a unified framework for applying score-based generative models to audio, detailing design principles and offering an open-source codebase, AudioDiffuser. The work demonstrates improved efficiency and quality across unconditional audio generation, speech enhancement, and text-to-speech synthesis.

Key insights

  • The Elucidated Diffusion Model (EDM) framework's design principles, including preconditioning and loss weighting, effectively transfer from image generation to audio, improving training stability and sample quality.
  • Higher-order deterministic samplers (e.g., DPM-Solver++, UniPC) significantly enhance sampling efficiency in audio tasks, achieving competitive or superior generation quality with substantially fewer network function evaluations.
  • Flexible conditioning mechanisms, such as in-context feature fusion and adaptive layer normalization, are crucial for achieving robust performance in conditional audio generation tasks like speech enhancement and text-to-speech.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.