Reproduce “Training-Efficient Text-to-Music Generation with State-Space Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wei-Jaw Lee, Fang-Chih Hsieh, Xuanjun Chen, Fang-Duo Tsai, Yi-Hsuan Yang

This research introduces a training-efficient, open-source framework for text-to-music generation utilizing State-Space Models (SSMs) and a two-stage hybrid design. The proposed Prefix SiMBA/Diffusion model achieves performance comparable to larger, proprietary-data-trained benchmarks while using approximately 10 times fewer FLOPs and 2% of the training data, all on a single consumer-grade GPU.

Key insights

  • SSM-based architectures, particularly Prefix SiMBA with lightweight channel mixing, demonstrate superior training efficiency and faster convergence than Transformer baselines for text-to-music generation.
  • The two-stage SSM/diffusion hybrid design effectively leverages the coarse-to-fine hierarchy of audio tokens, significantly enhancing generation quality and efficiency compared to single-stage models.
  • SSMs exhibit strong parameter efficiency, maintaining competitive perceptual quality and semantic alignment even when model size is scaled down significantly (e.g., to one-fourth of parameters).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.