Reproduce “TUTTI: Toward generalizable audio-to-score transcription via fully synthesized data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jianhuai Hu, Yashan Wang, Shangda Wu, Zhancheng Guo, Shijie Liang, Wuna Meng, Chuanqi Yang, Xiaobing Li and 2 more

TUTTI, a Transformer-based model, achieves generalizable audio-to-score transcription by leveraging TuttiCorpus, a massive, fully synthesized multi-instrumentation dataset comprising over 363,000 unique audio-score pairs. The model sets new state-of-the-art results across various A2S tasks and demonstrates robust transferability to instruments not present in its pre-training data.

Key insights

  • Large-scale synthetic data can effectively overcome the data scarcity bottleneck, enabling a generalizable A2S model without requiring specialized real-world annotated data.
  • A standard Transformer encoder-decoder architecture, when pre-trained on diverse synthetic data, can achieve state-of-the-art A2S performance, challenging the necessity of customized CNN acoustic front-ends.
  • Multi-instrumentation pre-training significantly enhances a model's representational capability, leading to improved performance and cross-instrument transferability even for instruments not encountered during training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.