Reproduce “EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis” on OpenResearch

About this paper

Ge Zhu, Yutong Wen, Marc-André Carbonneau, Zhiyao Duan

Researchers from the University of Rochester and Ubisoft La Forge developed EDMSound, an end-to-end diffusion model operating directly on complex spectrograms to generate high-quality audio with improved efficiency. The model achieved a state-of-the-art FAD score of 4.56 on the DCASE2023 Task 7 benchmark using only 50 sampling steps, and revealed "stitching copies" as a form of content imitation.

Key insights

• Directly modeling complex spectrograms, which include both magnitude and phase information, in an end-to-end system avoids fidelity losses often associated with cascaded audio generation approaches.
• Employing high-order Exponential Integrator (EI) based ODE solvers for the diffusion process allows for dramatically faster inference, requiring fewer neural function evaluations to achieve high-quality audio.
• Diffusion models for audio tend to create "stitching copies" by combining learned sound timbres from the training data with new temporal patterns, rather than producing exact duplicates.

Summary by alphaXiv.