Reproduce “A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis” on OpenResearch

About this paper

Liwei Lin, Qiuqiang Kong, Junyan Jiang, Gus Xia

Researchers at NYU Shanghai's Music X Lab and ByteDance developed a unified computational model that performs music source separation, automatic music transcription, and audio synthesis within a single framework. The model leverages explicit pitch-timbre disentanglement, allowing it to achieve zero-shot performance on instruments not seen during training and successfully synthesize new audio from learned timbres.

Key insights

• Explicitly learning disentangled representations of pitch and timbre acts as a powerful inductive bias, enabling better generalization across tasks and zero-shot performance.
• A Query-by-Example (QBE) network allows the model to perform separation and transcription for instruments not encountered during training.
• A novel Pitch-Translation Invariance Loss (L_PTI) specifically trains the timbre extraction module to produce representations invariant to pitch changes, which is crucial for successful synthesis and cleaner separation.

Summary by alphaXiv.