Reproduce “PianoVAM: A Multimodal Piano Performance Dataset” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yonghyun Kim, Junhyung Park, Joonhyung Bae, Kirak Kim, Taegyun Kwon, Alexander Lerch, Juhan Nam
PIANOVAM is a comprehensive multimodal dataset for piano performance, integrating synchronized top-view video, high-quality audio, accurate MIDI, hand landmarks, and semi-automated fingering labels. The dataset enables research into detailed performance analysis and demonstrates improved robustness in piano transcription, particularly under noisy and reverberant acoustic conditions.
Key insights
- Visual information from synchronized video effectively enhances the robustness and precision of piano transcription models, particularly when dealing with challenging acoustic conditions like noise or reverberation.
- A semi-automated annotation pipeline, merging computer vision techniques with human refinement, offers a reliable and scalable method for generating complex, labor-intensive labels such as piano fingering for large datasets.
- The integrated multimodal data in PIANOVAM provides a rich resource for investigating the intricate relationships between a pianist's physical movements and their musical output, fostering advancements in expressive performance analysis and performer-instrument interaction studies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.