Reproduce “Music-JEPA: Learning a World Model of Sound from Action” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyu Wang, Kun Fang, Yann LeCun
Music-JEPA predicts a piano sound’s next latent representation from the current audio and time-aligned note and pedal controls. Trained offline on paired piano audio and MIDI, it models how performance actions relate to sound rather than reconstructing waveforms. In temporal-shift tests, mismatch win rates are 0.929 and 0.991, versus 0.787 and 0.576 for AO-JEPA; results are limited to piano.
Key insights
- Explicitly modeling action-conditioned dynamics significantly enhances the learning of temporal relationships in music, enabling the model to accurately distinguish correct state transitions under various perturbations.
- The JEPA framework, by predicting latent representations instead of raw observations, effectively discards irrelevant details and focuses on the predictable, semantic aspects of high-dimensional audio, mimicking cognitive anticipation.
- The learned 'music world model' facilitates planning by allowing an inverse predictor to infer musical actions from consecutive latent states, demonstrating a pathway towards more interactive and goal-directed musical AI applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.