Reproduce “ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sang-Hoon Lee, Ha-Yeong Choi

The ReGen framework introduces a hierarchical multi-prompt representation generation approach and Generalized Flow Matching (GFM) for efficient waveform Diffusion Transformers. This method enhances speech intelligibility and speaker similarity in low-bitrate audio codecs and text-to-speech systems by explicitly disentangling semantic and acoustic features and mitigating over-smoothing.

Key insights

  • Shifting from representation alignment to a representation generation paradigm effectively addresses the "capacity mismatch" problem, allowing for explicit disentanglement of semantic and acoustic features and improving generative capacity for high-fidelity waveform synthesis.
  • A hierarchical multi-prompt mechanism, leveraging information from different representation levels (semantic, acoustic, waveform), provides robust guidance for the generation process, leading to improved speech intelligibility and speaker similarity.
  • Repulsive regularization in the vector field space via GFM is crucial for preventing mode collapse and maintaining high-frequency details in waveform generation, which is challenged by the multi-modal nature and phase inconsistencies of audio.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.