Reproduce “MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fang-Duo Tsai, Shih-Lun Wu, Weijaw Lee, Sheng-Ping Yang, Bo-Rui Chen, Hao-Chung Cheng, Yi-Hsuan Yang
MuseControlLite from National Taiwan University presents a lightweight fine-tuning mechanism for Diffusion Transformer-based text-to-music models, enabling precise, time-varying control over musical attributes and integrating audio inpainting/outpainting capabilities. The approach achieves competitive or superior control and audio quality with 6.75 times fewer trainable parameters than state-of-the-art ControlNet-based methods.
Key insights
- Rotary Positional Embeddings (ROPE) are crucial for accurately controlling time-varying conditions in controllable music generation, significantly improving melody accuracy from 10.7% to 58.6%.
- A decoupled cross-attention adapter design, inspired by IP-Adapter, effectively enables fine-grained control with significantly fewer trainable parameters than ControlNet-style models.
- A unified framework can effectively handle both time-varying musical attributes (melody, rhythm, dynamics) and reference audio signals (inpainting/outpainting) simultaneously.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.