Reproduce “PianoBench: Evaluating Musical Perception and Reasoning in Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Louis Bradshaw, Irmak Bukey, Daniel Chenyu Lin, John Thickstun, Chris Donahue
PianoBench tests localized musical perception and reasoning with 250 questions from 162 instructional piano videos. It compares piano audio and MIDI-text note-event descriptions against no-music controls. Open-weight audio models generally gain little over controls, while GPT 6 Astra reaches 93.2% on MIDI-text, 36.4 percentage points above its control. Above-chance controls and the benchmark’s limited scope constrain conclusions about general musical ability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.