Reproduce “LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huan Zhang, Vincent Cheung, Hayato Nishioka, Simon Dixon, Shinichi Furuya
LLaQo is an audio-language model designed to serve as a query-based coach for expressive music performance assessment, providing detailed, formative feedback. The model leverages an Audio-MAE encoder, a Q-former, and Vicuna-7b, trained on a comprehensive instruction-tuned dataset including the new NeuroPiano dataset, and demonstrates superior performance in generating relevant textual feedback and predicting teacher ratings compared to existing audio-language baselines.
Key insights
- Instruction-tuning with rich, diverse music performance assessment data is crucial for adapting general audio-language models to provide nuanced, domain-specific feedback.
- The integration of a specialized audio encoder (Audio-MAE) with an LLM via a Q-former effectively bridges the gap between low-level audio features and high-level linguistic interpretation for expressive performance.
- Query-based interaction enables targeted and personalized feedback, addressing specific learning needs and mimicking the pedagogical approach of human music coaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.