Reproduce “Diverse and Vivid Sound Generation from Text Descriptions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangwei Li, Xuenan Xu, Lingfeng Dai, Mengyue Wu, Kai Yu
Researchers at Shanghai Jiao Tong University developed a multi-stage framework for generating diverse and high-quality audio from free-form text descriptions. This system, which integrates VQGAN for discrete spectrogram representation, a Transformer for text-conditioned generation, and a MelGAN vocoder, achieved improved objective metrics on AudioCaps, including a SPICE score of 0.096 and FID of 0.778, while human evaluations rated generated audio relevance at 6.79 and quality at 6.13 out of 10.
Key insights
- Utilizing discrete spectrogram codebooks through VQGAN effectively addresses the high dimensionality of audio, making it amenable to Transformer-based generation and reducing computational complexity.
- Pre-training text encoders with audio-text contrastive learning (CL'A'P) significantly enhances the effectiveness of text features for representing audio events, leading to superior generation quality and relevance.
- Providing a richer textual context through a sequence of text features (e.g., 30 features) enables the model to better distinguish sound sources and their temporal relationships, outperforming single-feature inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.